[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:01 dist-test-slave-zpvx krb5kdc[23954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:01 dist-test-slave-zpvx krb5kdc[23954](info): set up 2 sockets
Aug 12 11:11:01 dist-test-slave-zpvx krb5kdc[23954](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 11:11:02.305605 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:02 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:02.436625 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34999
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:02.553225 23975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:02.553473 23975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:02.553512 23975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:02.557763 23975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:02.557827 23975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:02.557848 23975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:02.557865 23975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:02.557905 23975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:02.563078 23975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:02.564085 23975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:02.565166 23975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:02.573690 23991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:02.573704 23975 server_base.cc:1034] running on GCE node
W20260812 11:11:02.573729 23988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:02.573693 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:02.574541 23975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:02.575667 23975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:02.576839 23975 hybrid_clock.cc:648] HybridClock initialized: now 1786533062576843 us; error 43 us; skew 500 ppm
Aug 12 11:11:02 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533062, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:02.583612 23975 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:02.584656 23975 webserver.cc:460] Webserver started at http://127.23.97.254:44637/ using document root <none> and password file <none>
I20260812 11:11:02.585244 23975 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:02.585309 23975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:02.585533 23975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:02.587657 23975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "743267a0ef9c407dbe22ee88d4347652"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "262b98c8deb5dbcd0690a8e913f6a7b8"
server_key_iv: "ea41444215ea8a65164bd7d3b0b3fbf1"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.588243 23975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "743267a0ef9c407dbe22ee88d4347652"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "262b98c8deb5dbcd0690a8e913f6a7b8"
server_key_iv: "ea41444215ea8a65164bd7d3b0b3fbf1"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.592021 23975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:11:02.594617 24002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:02.595803 23975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:02.595932 23975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "743267a0ef9c407dbe22ee88d4347652"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "262b98c8deb5dbcd0690a8e913f6a7b8"
server_key_iv: "ea41444215ea8a65164bd7d3b0b3fbf1"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.596035 23975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:02.616498 23975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:02.620031 23975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:02.620235 23975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:02.628074 23975 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34999
I20260812 11:11:02.628099 24105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34999 every 8 connection(s)
I20260812 11:11:02.629086 23975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:02.632041 24110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:02.634500 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 23975
I20260812 11:11:02.634675 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:02.636396 23943 external_mini_cluster.cc:1428] Setting key 0c01b2e2f49ff1e72cba82c339dc8d92
I20260812 11:11:02.639331 24110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652: Bootstrap starting.
I20260812 11:11:02.641923 24110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:02.642663 24110 log.cc:824] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:02.644639 24110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652: No bootstrap required, opened a new log
I20260812 11:11:02.648525 24110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } }
I20260812 11:11:02.648744 24110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:02.648806 24110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743267a0ef9c407dbe22ee88d4347652, State: Initialized, Role: FOLLOWER
I20260812 11:11:02.649237 24110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } }
I20260812 11:11:02.649366 24110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:02.649458 24110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:02.649581 24110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:02.650454 24110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } }
I20260812 11:11:02.650804 24110 leader_election.cc:302] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743267a0ef9c407dbe22ee88d4347652; no voters:
I20260812 11:11:02.651108 24110 leader_election.cc:288] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:02.651206 24118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:02.651450 24118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [term 1 LEADER]: Becoming Leader. State: Replica: 743267a0ef9c407dbe22ee88d4347652, State: Running, Role: LEADER
I20260812 11:11:02.651757 24118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } }
I20260812 11:11:02.652590 24110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:02.653010 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743267a0ef9c407dbe22ee88d4347652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } } }
I20260812 11:11:02.653141 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:02.653937 24122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743267a0ef9c407dbe22ee88d4347652. Latest consensus state: current_term: 1 leader_uuid: "743267a0ef9c407dbe22ee88d4347652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743267a0ef9c407dbe22ee88d4347652" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34999 } } }
I20260812 11:11:02.654062 24122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:02.654373 24126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 11:11:02 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:02.658285 24126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:02.664191 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:02.640543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.23.97.254:34999)
0812 11:11:02.641057 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:02.641085 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:02.642124 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 11:11:02.645687 (+ 3563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:02.645704 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:02.646018 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:02.646353 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:02.648597 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.649922 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.651493 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.652142 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.652353 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:02.657685 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:02.657691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:02.657706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:02.657738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:02.659961 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:02.660664 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:02.660670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:02.660676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:02.660825 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:02.661163 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:02.661166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:02.661168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:02.661748 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:02.661960 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:02.662449 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 11:11:02.663466 (+ 1017us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":157,"threads_started":1}
I20260812 11:11:02.664849 24126 catalog_manager.cc:1347] Generated new cluster ID: 40963069d58342e7a42d2cc90f78b1cd
I20260812 11:11:02.664907 24126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:02.677001 24126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:02.677870 24126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:02.697706 24126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743267a0ef9c407dbe22ee88d4347652: Generated new TSK 0
I20260812 11:11:02.698566 24126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:02.733526 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:02.887609 24151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:02.887884 24151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:02.887959 24151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:02.892143 24151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:02.892226 24151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:02.892316 24151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:02.897943 24151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:02.898959 24151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:02.900122 24151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:02.910748 24169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:02.910912 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:02.910992 24151 server_base.cc:1034] running on GCE node
W20260812 11:11:02.910862 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:02.911487 24151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:02.912078 24151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:02.913244 24151 hybrid_clock.cc:648] HybridClock initialized: now 1786533062913200 us; error 54 us; skew 500 ppm
Aug 12 11:11:02 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533062, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:02.916685 24151 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:02.917469 24151 webserver.cc:460] Webserver started at http://127.23.97.193:45659/ using document root <none> and password file <none>
I20260812 11:11:02.917979 24151 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:02.918037 24151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:02.918197 24151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:02.920594 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "c58fd41b8ed042a0b594066945fa000c"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "0eaf2888e77cd6bae9d6f5ae282bd4b4"
server_key_iv: "8909ba1b76db95d8b28a54539e8bf38f"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.921164 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "c58fd41b8ed042a0b594066945fa000c"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "0eaf2888e77cd6bae9d6f5ae282bd4b4"
server_key_iv: "8909ba1b76db95d8b28a54539e8bf38f"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.925125 24151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:11:02.927639 24184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:02.928967 24151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:11:02.929116 24151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "c58fd41b8ed042a0b594066945fa000c"
format_stamp: "Formatted at 2026-08-12 11:11:02 on dist-test-slave-zpvx"
server_key: "0eaf2888e77cd6bae9d6f5ae282bd4b4"
server_key_iv: "8909ba1b76db95d8b28a54539e8bf38f"
server_key_version: "encryptionkey@0"
I20260812 11:11:02.929243 24151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:02.950901 24151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:02.954665 24151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:02.954879 24151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:02.955549 24151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:02.956482 24151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:02.956550 24151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:02.956610 24151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:02.956655 24151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:02.976490 24151 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44623
I20260812 11:11:02.976598 24428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44623 every 8 connection(s)
I20260812 11:11:02.977450 24151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:02.981315 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24151
I20260812 11:11:02.981436 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:02.981716 23943 external_mini_cluster.cc:1428] Setting key 248502a2cd56fc90c3fcdf840201fe9e
Aug 12 11:11:02 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533062, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:02.994164 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:02.979538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:51827 (local address 127.23.97.254:34999)
0812 11:11:02.979677 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:02.979681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:02.980770 (+ 1089us) server_negotiation.cc:407] Connection header received
0812 11:11:02.982644 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:02.982647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:02.982697 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:02.982792 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:02.984301 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.985208 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.986524 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.986772 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.986881 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:02.990170 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:02.990173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:02.990175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:02.990200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:02.991756 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:02.992492 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:02.992494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:02.992495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:02.992573 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:02.992991 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:02.992994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:02.992995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:02.993169 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:02.993250 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:02.993925 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 11:11:02.994033 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:02.995383 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:02.979923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34999 (local address 127.23.97.193:51827)
0812 11:11:02.980603 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:02.980652 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:02.982386 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:02.982966 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:02.982973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:02.983519 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:02.984139 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:02.984155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.985341 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.985344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:02.986393 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:02.986403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:02.986933 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:02.986937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:02.987133 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:02.987830 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:02.987846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:02.989970 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:02.991900 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:02.991910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:02.991928 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:02.992363 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:02.992675 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:02.992681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:02.992684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:02.992890 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:02.993283 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:02.993290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:02.993645 (+ 355us) client_negotiation.cc:769] Sending connection context
0812 11:11:02.993886 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 11:11:02.994269 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":160,"threads_started":1}
I20260812 11:11:02.996873 24431 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34999
I20260812 11:11:02.997133 24431 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:02.997654 24431 heartbeater.cc:511] Master 127.23.97.254:34999 requested a full tablet report, sending...
I20260812 11:11:02.999822 24038 ts_manager.cc:194] Registered new tserver with Master: c58fd41b8ed042a0b594066945fa000c (127.23.97.193:44623)
I20260812 11:11:03.001250 24038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:51827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:03.013434 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:03.157699 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:03.158002 24439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:03.158077 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:03.162472 24439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:03.162571 24439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:03.162665 24439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:03.168815 24439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:03.169965 24439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:03.171147 24439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:03.182269 24445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:03.182368 24439 server_base.cc:1034] running on GCE node
W20260812 11:11:03.182245 24449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:03.182246 24452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:03.182876 24439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:03.183504 24439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:03.184689 24439 hybrid_clock.cc:648] HybridClock initialized: now 1786533063184680 us; error 30 us; skew 500 ppm
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:03.187980 24439 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:03.188930 24439 webserver.cc:460] Webserver started at http://127.23.97.194:43125/ using document root <none> and password file <none>
I20260812 11:11:03.189507 24439 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:03.189601 24439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:03.189845 24439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:03.192010 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "8c1db2faaddd423bb48ba751478faccc"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "6c0cb188dff4668bc56e0739091d848b"
server_key_iv: "2853de61a9d6532680c95dd90c70e0b6"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.192651 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "8c1db2faaddd423bb48ba751478faccc"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "6c0cb188dff4668bc56e0739091d848b"
server_key_iv: "2853de61a9d6532680c95dd90c70e0b6"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.196765 24439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:11:03.199250 24464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.200691 24439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:11:03.200814 24439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "8c1db2faaddd423bb48ba751478faccc"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "6c0cb188dff4668bc56e0739091d848b"
server_key_iv: "2853de61a9d6532680c95dd90c70e0b6"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.200924 24439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:03.228627 24439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:03.231982 24439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:03.232288 24439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:03.233245 24439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:03.234334 24439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:03.234395 24439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.234463 24439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:03.234503 24439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.252641 24439 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:39883
I20260812 11:11:03.252770 24676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:39883 every 8 connection(s)
I20260812 11:11:03.253784 24439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:03.261255 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24439
I20260812 11:11:03.261380 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:03.261693 23943 external_mini_cluster.cc:1428] Setting key 46269ba2f5de4ca1ef442d132337aea1
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:03.268833 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.255823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:53981 (local address 127.23.97.254:34999)
0812 11:11:03.255983 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:03.255988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:03.257085 (+ 1097us) server_negotiation.cc:407] Connection header received
0812 11:11:03.258329 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:03.258332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:03.258380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:03.258457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:03.259764 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.260273 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.261569 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.261883 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.261997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.264715 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:03.264721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:03.264727 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:03.264760 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:03.266467 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.267132 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.267134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.267136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.267210 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.267602 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.267604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.267605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.267774 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:03.267849 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:03.268545 (+ 696us) server_negotiation.cc:299] Negotiation successful
0812 11:11:03.268688 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:11:03.270107 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.256281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34999 (local address 127.23.97.194:53981)
0812 11:11:03.256919 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:03.256963 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:03.258124 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:03.258694 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:03.258701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:03.259139 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:03.259622 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:03.259636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.260489 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.260493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:03.261433 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:03.261447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.262023 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.262026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:03.262245 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.262861 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:03.262880 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:03.264545 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:03.266588 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:03.266597 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:03.266615 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:03.267024 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:03.267299 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:03.267306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:03.267310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:03.267502 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:03.267880 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:03.267890 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:03.268218 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 11:11:03.268491 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 11:11:03.268874 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":197,"threads_started":1}
I20260812 11:11:03.271421 24677 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34999
I20260812 11:11:03.271677 24677 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:03.272178 24677 heartbeater.cc:511] Master 127.23.97.254:34999 requested a full tablet report, sending...
I20260812 11:11:03.273365 24038 ts_manager.cc:194] Registered new tserver with Master: 8c1db2faaddd423bb48ba751478faccc (127.23.97.194:39883)
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:11:03.274345 24038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:53981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:03.295629 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:03.438978 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:03.439248 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:03.439333 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:03.443360 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:03.443420 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:03.443519 24687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:03.448810 24687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:03.449918 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:03.451020 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:03.460934 24712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:03.461148 24687 server_base.cc:1034] running on GCE node
W20260812 11:11:03.460950 24706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:03.460973 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:03.461767 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:03.462504 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:03.463665 24687 hybrid_clock.cc:648] HybridClock initialized: now 1786533063463647 us; error 33 us; skew 500 ppm
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:03.466804 24687 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:03.467689 24687 webserver.cc:460] Webserver started at http://127.23.97.195:40673/ using document root <none> and password file <none>
I20260812 11:11:03.468283 24687 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:03.468379 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:03.468655 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:03.470772 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "128e874ffaf348b9a1d976bccb4ae771"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "99f3088d60df597981e764613a8ae2f3"
server_key_iv: "a79568b8571c21065aa3ecb522ae8126"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.471371 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "128e874ffaf348b9a1d976bccb4ae771"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "99f3088d60df597981e764613a8ae2f3"
server_key_iv: "a79568b8571c21065aa3ecb522ae8126"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.475262 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 11:11:03.477805 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.479015 24687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:03.479156 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "128e874ffaf348b9a1d976bccb4ae771"
format_stamp: "Formatted at 2026-08-12 11:11:03 on dist-test-slave-zpvx"
server_key: "99f3088d60df597981e764613a8ae2f3"
server_key_iv: "a79568b8571c21065aa3ecb522ae8126"
server_key_version: "encryptionkey@0"
I20260812 11:11:03.479279 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:03.490968 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:03.494194 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:03.494402 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:03.495005 24687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:03.495841 24687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:03.495914 24687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.495987 24687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:03.496031 24687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:03.505652 24687 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:37269
I20260812 11:11:03.505697 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:37269 every 8 connection(s)
I20260812 11:11:03.506639 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:03.513233 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24687
I20260812 11:11:03.513382 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:03.513662 23943 external_mini_cluster.cc:1428] Setting key b3d922a74af57353abcd4e4b10a0c8d9
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:03.521174 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.508768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:51355 (local address 127.23.97.254:34999)
0812 11:11:03.508916 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:03.508922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:03.509846 (+ 924us) server_negotiation.cc:407] Connection header received
0812 11:11:03.511107 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:03.511109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:03.511159 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:03.511261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:03.512488 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.513075 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.514435 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.514763 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.514873 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.517538 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:03.517541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:03.517543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:03.517568 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:03.519071 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.519569 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.519572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.519573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.519660 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.520122 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.520125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.520127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.520324 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:03.520485 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:03.520952 (+ 467us) server_negotiation.cc:299] Negotiation successful
0812 11:11:03.521054 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:11:03.521988 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.509233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34999 (local address 127.23.97.195:51355)
0812 11:11:03.509688 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:03.509733 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:03.510909 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:03.511439 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:03.511445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:03.511807 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:03.512286 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:03.512300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.513196 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.513199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:03.514330 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:03.514340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.514912 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.514915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:03.515081 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.515710 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:03.515730 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:03.517393 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:03.519184 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:03.519188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:03.519200 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:03.519458 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:03.519784 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:03.519789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:03.519792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:03.519959 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:03.520505 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:03.520510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:03.520739 (+ 229us) client_negotiation.cc:769] Sending connection context
0812 11:11:03.520897 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 11:11:03.521135 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260812 11:11:03.523177 24956 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34999
I20260812 11:11:03.523466 24956 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:03.523972 24956 heartbeater.cc:511] Master 127.23.97.254:34999 requested a full tablet report, sending...
I20260812 11:11:03.525094 24038 ts_manager.cc:194] Registered new tserver with Master: 128e874ffaf348b9a1d976bccb4ae771 (127.23.97.195:37269)
I20260812 11:11:03.525665 24038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:51355
I20260812 11:11:03.528254 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:03.556416 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.546775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.23.97.193:44623)
0812 11:11:03.547080 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:03.547088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:03.547116 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:11:03.547224 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:03.547231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:03.547428 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:03.547588 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:03.548287 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.549108 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.549989 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.550371 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.550494 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.552867 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:03.552876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:03.552891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:03.552914 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:03.554572 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.555033 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.555038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.555039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.555117 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.555403 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.555407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.555409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.555655 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:03.555869 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:03.556049 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:11:03.556214 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
W20260812 11:11:03.557407 24238 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[23954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:03.585433 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:03.576615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.23.97.193:44623)
0812 11:11:03.576790 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:03.576795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:03.576919 (+ 124us) server_negotiation.cc:407] Connection header received
0812 11:11:03.577103 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:03.577108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:03.577166 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:03.577256 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:03.578063 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.578584 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.579420 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:03.579636 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:03.579719 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:03.582252 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:03.582256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:03.582258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:03.582282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:03.583762 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.584232 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.584235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.584236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.584311 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:03.584611 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:03.584614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:03.584615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:03.584777 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:03.584880 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:03.585094 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 11:11:03.585254 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:11:03.586565 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24151
I20260812 11:11:03.602398 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24439
I20260812 11:11:03.615868 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24687
I20260812 11:11:03.631898 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 23975
2026-08-12T11:11:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2659 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[24979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[24979](info): set up 2 sockets
Aug 12 11:11:03 dist-test-slave-zpvx krb5kdc[24979](info): commencing operation
krb5kdc: starting...
W20260812 11:11:04.943923 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:05.067556 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35027
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:05.177184 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:05.177453 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:05.177524 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:05.181504 24997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:05.181581 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:05.181617 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:05.181674 24997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:05.181715 24997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:05.186759 24997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:05.187732 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:05.188783 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:05.197168 25009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.197178 25008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.197178 25012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:05.197403 24997 server_base.cc:1034] running on GCE node
I20260812 11:11:05.197994 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:05.199018 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:05.200258 24997 hybrid_clock.cc:648] HybridClock initialized: now 1786533065200228 us; error 42 us; skew 500 ppm
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:05.203049 24997 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:05.203905 24997 webserver.cc:460] Webserver started at http://127.23.97.254:35797/ using document root <none> and password file <none>
I20260812 11:11:05.204479 24997 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:05.204546 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:05.204711 24997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:05.206627 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "bd60d9e1c500449198657588ddfcc12b"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2014506011de7b48f9d2eff4040d8f99"
server_key_iv: "a60054abe7577a7880ec709da9daf537"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.207129 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "bd60d9e1c500449198657588ddfcc12b"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2014506011de7b48f9d2eff4040d8f99"
server_key_iv: "a60054abe7577a7880ec709da9daf537"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.210731 24997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:05.213095 25022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.214327 24997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:05.214453 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "bd60d9e1c500449198657588ddfcc12b"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2014506011de7b48f9d2eff4040d8f99"
server_key_iv: "a60054abe7577a7880ec709da9daf537"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.214557 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:05.233963 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:05.236929 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:05.237125 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:05.244335 24997 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:35027
I20260812 11:11:05.244360 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:35027 every 8 connection(s)
I20260812 11:11:05.245401 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:05.248101 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:05.253255 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24997
I20260812 11:11:05.253378 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:05.253639 23943 external_mini_cluster.cc:1428] Setting key 0a3e7a4a3bf45162d3f8c5de2e27a5b3
I20260812 11:11:05.253865 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b: Bootstrap starting.
I20260812 11:11:05.256345 25130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:05.257128 25130 log.cc:824] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:05.259316 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b: No bootstrap required, opened a new log
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:05.263546 25130 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } }
I20260812 11:11:05.263796 25130 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:05.263868 25130 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd60d9e1c500449198657588ddfcc12b, State: Initialized, Role: FOLLOWER
I20260812 11:11:05.264272 25130 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } }
I20260812 11:11:05.264442 25130 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:05.264523 25130 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:05.264653 25130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:05.265455 25130 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } }
I20260812 11:11:05.265818 25130 leader_election.cc:302] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd60d9e1c500449198657588ddfcc12b; no voters:
I20260812 11:11:05.266201 25130 leader_election.cc:288] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:05.266261 25141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:05.266494 25141 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [term 1 LEADER]: Becoming Leader. State: Replica: bd60d9e1c500449198657588ddfcc12b, State: Running, Role: LEADER
I20260812 11:11:05.266806 25141 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } }
I20260812 11:11:05.267506 25130 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:05.268712 25143 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60d9e1c500449198657588ddfcc12b. Latest consensus state: current_term: 1 leader_uuid: "bd60d9e1c500449198657588ddfcc12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } } }
I20260812 11:11:05.268846 25143 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:05.269152 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.254999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.23.97.254:35027)
0812 11:11:05.255442 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:05.255458 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:05.255493 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:05.256098 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:05.256114 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:05.256522 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:05.256847 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:05.257708 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.259142 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.260054 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.260768 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.260941 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.263649 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:05.263654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:05.263667 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:05.263691 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:05.265683 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.266190 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.266196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.266202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.266377 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.266774 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.266778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.266779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.267261 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:05.267452 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:05.267791 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 11:11:05.268061 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260812 11:11:05.269467 25156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:05.269148 25142 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd60d9e1c500449198657588ddfcc12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60d9e1c500449198657588ddfcc12b" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35027 } } }
I20260812 11:11:05.269752 25142 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:05.272617 25156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:05.278214 25156 catalog_manager.cc:1347] Generated new cluster ID: b181fab141e74a949f053e872a0f460a
I20260812 11:11:05.278285 25156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:05.286293 25156 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:05.287149 25156 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:05.291426 25156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd60d9e1c500449198657588ddfcc12b: Generated new TSK 0
I20260812 11:11:05.292027 25156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:05.323534 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:05.435523 25182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:05.435767 25182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:05.435809 25182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:05.439714 25182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:05.439764 25182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:05.439826 25182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:05.444758 25182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:05.445648 25182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:05.446563 25182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:05.455798 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.455798 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.455801 25200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:05.455989 25182 server_base.cc:1034] running on GCE node
I20260812 11:11:05.456578 25182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:05.457178 25182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:05.458366 25182 hybrid_clock.cc:648] HybridClock initialized: now 1786533065458356 us; error 32 us; skew 500 ppm
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:05.461198 25182 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:05.462074 25182 webserver.cc:460] Webserver started at http://127.23.97.193:43765/ using document root <none> and password file <none>
I20260812 11:11:05.462631 25182 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:05.462723 25182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:05.462956 25182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:05.465068 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "4b076851d96b4de7bb94655a09b884cb"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2198054fd99d912c4c79fae247654ba4"
server_key_iv: "932e40850c8c35dbeb95bc4c9be4dbc8"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.465632 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "4b076851d96b4de7bb94655a09b884cb"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2198054fd99d912c4c79fae247654ba4"
server_key_iv: "932e40850c8c35dbeb95bc4c9be4dbc8"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.469498 25182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:11:05.471926 25215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.473091 25182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:05.473233 25182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "4b076851d96b4de7bb94655a09b884cb"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "2198054fd99d912c4c79fae247654ba4"
server_key_iv: "932e40850c8c35dbeb95bc4c9be4dbc8"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.473340 25182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:05.497284 25182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:05.503998 25182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:05.504227 25182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:05.504858 25182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:05.505710 25182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:05.505785 25182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.505861 25182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:05.505904 25182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.515061 25182 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:41819
I20260812 11:11:05.515090 25427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:41819 every 8 connection(s)
I20260812 11:11:05.515913 25182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:05.519814 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25182
I20260812 11:11:05.519935 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:05.520215 23943 external_mini_cluster.cc:1428] Setting key 0bb22f65f3b7bb066653d0c86d4f618e
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:05.530275 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.517744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:56623 (local address 127.23.97.254:35027)
0812 11:11:05.517879 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:05.517883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:05.518773 (+ 890us) server_negotiation.cc:407] Connection header received
0812 11:11:05.519688 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:05.519691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:05.519761 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:05.519846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:05.521359 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.522092 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.523212 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.523623 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.523757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.526688 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:05.526691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:05.526693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:05.526717 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:05.528201 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.528756 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.528761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.528763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.528845 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.529212 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.529216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.529218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.529392 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:05.529503 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:05.529980 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 11:11:05.530102 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:05.531024 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.518128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35027 (local address 127.23.97.193:56623)
0812 11:11:05.518632 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:05.518675 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:05.519482 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:05.520063 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:05.520072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:05.520550 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:05.521202 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.521231 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.522226 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.522229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.523080 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.523090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.523839 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.523844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.524066 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.524706 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:05.524740 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:05.526507 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:05.528324 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.528330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.528344 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.528646 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.528940 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.528944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.528946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.529103 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.529502 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:05.529509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:05.529742 (+ 233us) client_negotiation.cc:769] Sending connection context
0812 11:11:05.529921 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 11:11:05.530240 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260812 11:11:05.532163 25429 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35027
I20260812 11:11:05.532513 25429 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:05.533118 25429 heartbeater.cc:511] Master 127.23.97.254:35027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:05.534641 25064 ts_manager.cc:194] Registered new tserver with Master: 4b076851d96b4de7bb94655a09b884cb (127.23.97.193:41819)
I20260812 11:11:05.535746 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:56623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:05.548955 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:05.659379 25435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:05.659611 25435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:05.659652 25435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:05.663614 25435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:05.663666 25435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:05.663724 25435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:05.668921 25435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:05.669931 25435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:05.670904 25435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:05.680140 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.680152 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.680979 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:05.681186 25435 server_base.cc:1034] running on GCE node
I20260812 11:11:05.681752 25435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:05.682453 25435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:05.683630 25435 hybrid_clock.cc:648] HybridClock initialized: now 1786533065683550 us; error 96 us; skew 500 ppm
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:05.687422 25435 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:05.690940 25435 webserver.cc:460] Webserver started at http://127.23.97.194:39877/ using document root <none> and password file <none>
I20260812 11:11:05.691808 25435 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:05.691922 25435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:05.692232 25435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:05.695008 25435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "a1be81569b3c40b18febbe806036b018"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "51cde4577f2c5057157fde4b468287b0"
server_key_iv: "94c544de74e4eada7016e5ade316f629"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.695762 25435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "a1be81569b3c40b18febbe806036b018"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "51cde4577f2c5057157fde4b468287b0"
server_key_iv: "94c544de74e4eada7016e5ade316f629"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.700798 25435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 11:11:05.704567 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.706501 25435 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:11:05.706640 25435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "a1be81569b3c40b18febbe806036b018"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "51cde4577f2c5057157fde4b468287b0"
server_key_iv: "94c544de74e4eada7016e5ade316f629"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.706770 25435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:05.730142 25435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:05.733482 25435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:05.733688 25435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:05.734273 25435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:05.735100 25435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:05.735179 25435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.735258 25435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:05.735311 25435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.744421 25435 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:43875
I20260812 11:11:05.744436 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:43875 every 8 connection(s)
I20260812 11:11:05.745280 25435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:05.747206 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25435
I20260812 11:11:05.747336 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:05.747620 23943 external_mini_cluster.cc:1428] Setting key 7be7ce7d55067a7d3f55f4616ca8ad9a
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:11:05.760054 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.747020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:33509 (local address 127.23.97.254:35027)
0812 11:11:05.747177 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:05.747184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:05.748184 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 11:11:05.749410 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:05.749415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:05.749464 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:05.749557 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:05.750643 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.751444 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.753097 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.753504 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.753631 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.756434 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:05.756441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:05.756446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:05.756480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:05.757888 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.758511 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.758514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.758515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.758594 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.758976 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.758979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.758980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.759198 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:05.759278 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:05.759801 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 11:11:05.759899 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:11:05.760922 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.747480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35027 (local address 127.23.97.194:33509)
0812 11:11:05.748024 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:05.748081 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:05.749188 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:05.749709 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:05.749723 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:05.750054 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:05.750498 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.750511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.751570 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.751576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.752976 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.752986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.753652 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.753655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.753793 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.754367 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:05.754383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:05.756200 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:05.758013 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.758020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.758036 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.758404 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.758695 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.758698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.758700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.758884 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.759304 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:05.759312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:05.759586 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 11:11:05.759753 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 11:11:05.759991 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:11:05.762059 25687 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35027
I20260812 11:11:05.762322 25687 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:05.762804 25687 heartbeater.cc:511] Master 127.23.97.254:35027 requested a full tablet report, sending...
I20260812 11:11:05.763890 25064 ts_manager.cc:194] Registered new tserver with Master: a1be81569b3c40b18febbe806036b018 (127.23.97.194:43875)
I20260812 11:11:05.764500 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:33509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:05.777257 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:05.892050 25694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:05.892277 25694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:05.892319 25694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:05.896196 25694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:05.896286 25694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:05.896342 25694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:05.901710 25694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:05.902673 25694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:05.903595 25694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:05.912891 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.912911 25706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:05.912901 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:05.913259 25694 server_base.cc:1034] running on GCE node
I20260812 11:11:05.913626 25694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:05.914232 25694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:05.915422 25694 hybrid_clock.cc:648] HybridClock initialized: now 1786533065915406 us; error 27 us; skew 500 ppm
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:05.918289 25694 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:05.919157 25694 webserver.cc:460] Webserver started at http://127.23.97.195:33401/ using document root <none> and password file <none>
I20260812 11:11:05.919704 25694 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:05.919806 25694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:05.920035 25694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:05.922103 25694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "09f2de53e052456b8f7eefaed87eed5f"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "de83dcbbf8497870153b3f38ec38c991"
server_key_iv: "b64107316e1b87424ec259a3dddf8243"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.922721 25694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "09f2de53e052456b8f7eefaed87eed5f"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "de83dcbbf8497870153b3f38ec38c991"
server_key_iv: "b64107316e1b87424ec259a3dddf8243"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.926333 25694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:05.928802 25726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.929956 25694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:05.930050 25694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "09f2de53e052456b8f7eefaed87eed5f"
format_stamp: "Formatted at 2026-08-12 11:11:05 on dist-test-slave-zpvx"
server_key: "de83dcbbf8497870153b3f38ec38c991"
server_key_iv: "b64107316e1b87424ec259a3dddf8243"
server_key_version: "encryptionkey@0"
I20260812 11:11:05.930128 25694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:05.957222 25694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:05.959882 25694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:05.960033 25694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:05.960565 25694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:05.961413 25694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:05.961462 25694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.961503 25694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:05.961524 25694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:05.970623 25694 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:43283
I20260812 11:11:05.970697 25946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:43283 every 8 connection(s)
I20260812 11:11:05.971487 25694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:05.973178 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25694
I20260812 11:11:05.973310 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:05.973616 23943 external_mini_cluster.cc:1428] Setting key f4a9f691d263525a3f111512c612e3bb
Aug 12 11:11:05 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:05.984314 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.973377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:47595 (local address 127.23.97.254:35027)
0812 11:11:05.973542 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:05.973547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:05.974485 (+ 938us) server_negotiation.cc:407] Connection header received
0812 11:11:05.975807 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:05.975810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:05.975854 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:05.975908 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:05.977087 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.977601 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.978733 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.978940 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.979018 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.981261 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:05.981264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:05.981266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:05.981288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:05.982594 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.983099 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.983102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.983103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.983173 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:05.983456 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:05.983458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:05.983459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:05.983613 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:05.983687 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:05.984110 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 11:11:05.984199 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:11:05.985172 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:05.973773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35027 (local address 127.23.97.195:47595)
0812 11:11:05.974332 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:05.974390 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:05.975589 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:05.976076 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:05.976082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:05.976437 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:05.976967 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.976981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.977705 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.977707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.978624 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:05.978634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:05.979064 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:05.979067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:05.979199 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:05.979676 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:05.979691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:05.981110 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:05.982702 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.982707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.982717 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.982991 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.983250 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:05.983252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:05.983254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:05.983380 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:05.983710 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:05.983714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:05.983916 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 11:11:05.984065 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:11:05.984326 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
I20260812 11:11:05.986239 25948 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35027
I20260812 11:11:05.986487 25948 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:05.986948 25948 heartbeater.cc:511] Master 127.23.97.254:35027 requested a full tablet report, sending...
I20260812 11:11:05.987915 25064 ts_manager.cc:194] Registered new tserver with Master: 09f2de53e052456b8f7eefaed87eed5f (127.23.97.195:43283)
I20260812 11:11:05.988494 25064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:47595
I20260812 11:11:05.998988 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:11:06.009691 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.002831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.23.97.254:35027)
0812 11:11:06.002945 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.002948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.003124 (+ 176us) server_negotiation.cc:407] Connection header received
0812 11:11:06.003226 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.003228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.003266 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.003325 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:06.003991 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.004494 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.005302 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.005498 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.005569 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.006912 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:06.006915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:06.006917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:06.006937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:06.008225 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.008655 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.008657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.008659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.008734 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.008964 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.008966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.008967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.009178 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:06.009280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.009480 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.009573 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:11:06.015053 25064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:06.018215 25064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:06.033525 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.027703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.23.97.195:43283)
0812 11:11:06.028111 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.028123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.028259 (+ 136us) server_negotiation.cc:407] Connection header received
0812 11:11:06.028492 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.028500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.028719 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.028908 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:06.029225 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.030261 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.031910 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.032771 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.032876 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.033074 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.033275 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.033378 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":168,"threads_started":1}
I20260812 11:11:06.033573 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.027588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43283 (local address 127.0.0.1:47062)
0812 11:11:06.028100 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:06.028152 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:06.028302 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:06.028915 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:06.028921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:06.028945 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:06.029122 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.029132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.030380 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.030385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.031784 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.031792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.032906 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.032911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.033062 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.033115 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 11:11:06.033225 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:11:06.033365 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260812 11:11:06.033572 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.028346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41819 (local address 127.0.0.1:52266)
0812 11:11:06.028602 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:06.028615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:06.028705 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:06.029848 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:06.029851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:06.029870 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:06.029956 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.029961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.031123 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.031126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.032179 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.032191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.033114 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.033118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.033208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.033237 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:11:06.033279 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 11:11:06.033412 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":25,"thread_start_us":133,"threads_started":1}
I20260812 11:11:06.034174 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.028570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.23.97.193:41819)
0812 11:11:06.029366 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.029383 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.029408 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:11:06.029482 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.029487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.029696 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.029857 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:06.030081 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.030971 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.032327 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.032963 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.033746 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.033906 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.033964 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.034055 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":97,"threads_started":1}
I20260812 11:11:06.035161 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.028704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43875 (local address 127.0.0.1:57738)
0812 11:11:06.028951 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:06.028964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:06.029038 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:06.030273 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:06.030278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:06.030297 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:06.030375 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.030381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.031784 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.031787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.032977 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.032988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.034808 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.034810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.034871 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.034885 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 11:11:06.034963 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 11:11:06.035032 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":27,"thread_start_us":95,"threads_started":1}
I20260812 11:11:06.035247 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.029262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.23.97.194:43875)
0812 11:11:06.029649 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.029660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.029682 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:11:06.029757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.029765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.029934 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.030098 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:06.030483 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.031621 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.033141 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.034547 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.034632 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.034721 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.035008 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.035098 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":108,"threads_started":1}
I20260812 11:11:06.036834 25828 tablet_service.cc:1467] Processing CreateTablet for tablet 8c246772f3f541afa922da23885527e4 (DEFAULT_TABLE table=test-table [id=7f86dd3c85f34d9fb77eae4d1340c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:06.036830 25306 tablet_service.cc:1467] Processing CreateTablet for tablet 8c246772f3f541afa922da23885527e4 (DEFAULT_TABLE table=test-table [id=7f86dd3c85f34d9fb77eae4d1340c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:06.037659 25551 tablet_service.cc:1467] Processing CreateTablet for tablet 8c246772f3f541afa922da23885527e4 (DEFAULT_TABLE table=test-table [id=7f86dd3c85f34d9fb77eae4d1340c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:06.037971 25828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c246772f3f541afa922da23885527e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:06.037964 25306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c246772f3f541afa922da23885527e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:06.038506 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c246772f3f541afa922da23885527e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:06.044345 25977 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Bootstrap starting.
I20260812 11:11:06.045809 25978 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Bootstrap starting.
I20260812 11:11:06.047076 25977 tablet_bootstrap.cc:654] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:06.047583 25978 tablet_bootstrap.cc:654] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:06.047842 25977 log.cc:824] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:06.048421 25978 log.cc:824] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:06.048532 25980 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Bootstrap starting.
I20260812 11:11:06.049922 25977 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: No bootstrap required, opened a new log
I20260812 11:11:06.050127 25977 ts_tablet_manager.cc:1397] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 11:11:06.050452 25978 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: No bootstrap required, opened a new log
I20260812 11:11:06.050643 25978 ts_tablet_manager.cc:1397] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:11:06.050806 25980 tablet_bootstrap.cc:654] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:06.051627 25980 log.cc:824] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:06.053321 25980 tablet_bootstrap.cc:492] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: No bootstrap required, opened a new log
I20260812 11:11:06.053507 25980 ts_tablet_manager.cc:1397] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:11:06.054203 25977 raft_consensus.cc:348] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.054486 25977 raft_consensus.cc:374] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:06.054543 25977 raft_consensus.cc:729] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1be81569b3c40b18febbe806036b018, State: Initialized, Role: FOLLOWER
I20260812 11:11:06.055003 25977 consensus_queue.cc:260] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.055192 25978 raft_consensus.cc:348] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.055446 25978 raft_consensus.cc:374] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:06.055503 25978 raft_consensus.cc:729] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b076851d96b4de7bb94655a09b884cb, State: Initialized, Role: FOLLOWER
I20260812 11:11:06.055809 25687 heartbeater.cc:503] Master 127.23.97.254:35027 was elected leader, sending a full tablet report...
I20260812 11:11:06.055953 25978 consensus_queue.cc:260] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.056627 25429 heartbeater.cc:503] Master 127.23.97.254:35027 was elected leader, sending a full tablet report...
I20260812 11:11:06.056939 25980 raft_consensus.cc:348] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.057201 25977 ts_tablet_manager.cc:1428] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260812 11:11:06.057197 25980 raft_consensus.cc:374] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:06.057263 25980 raft_consensus.cc:729] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f2de53e052456b8f7eefaed87eed5f, State: Initialized, Role: FOLLOWER
I20260812 11:11:06.057775 25980 consensus_queue.cc:260] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.057971 25978 ts_tablet_manager.cc:1428] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260812 11:11:06.058678 25980 ts_tablet_manager.cc:1428] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260812 11:11:06.059104 25948 heartbeater.cc:503] Master 127.23.97.254:35027 was elected leader, sending a full tablet report...
I20260812 11:11:06.184857 25992 raft_consensus.cc:482] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:06.185119 25992 raft_consensus.cc:504] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.186276 25992 leader_election.cc:288] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b076851d96b4de7bb94655a09b884cb (127.23.97.193:41819), a1be81569b3c40b18febbe806036b018 (127.23.97.194:43875)
I20260812 11:11:06.190508 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.186540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41819 (local address 127.23.97.195:44625)
0812 11:11:06.186690 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:06.186709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:06.186833 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:06.187101 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:06.187104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:06.187125 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:06.187259 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.187265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.188223 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.188226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.189203 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.189212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.190089 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.190093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.190200 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.190226 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:11:06.190305 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:11:06.190375 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 11:11:06.190488 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.186655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:44625 (local address 127.23.97.193:41819)
0812 11:11:06.186830 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.186835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.186854 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:06.186918 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.186921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.186979 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.187069 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:06.187394 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.188112 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.189341 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.189975 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.190084 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.190125 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.190331 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.190385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:11:06.191011 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.187010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:35917 (local address 127.23.97.194:43875)
0812 11:11:06.187144 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.187149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.187436 (+ 287us) server_negotiation.cc:407] Connection header received
0812 11:11:06.187536 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.187538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.187581 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.187644 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:06.187930 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.188603 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.189873 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.190443 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.190563 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.190611 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.190832 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.190905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:11:06.191013 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.186875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43875 (local address 127.23.97.195:35917)
0812 11:11:06.187305 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:06.187340 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:06.187444 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:06.187686 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:06.187689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:06.187707 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:06.187827 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.187833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.188720 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.188722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.189745 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:06.189755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.190596 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.190601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:06.190714 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.190744 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:11:06.190812 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 11:11:06.190871 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":141,"threads_started":1}
I20260812 11:11:06.191213 25344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c246772f3f541afa922da23885527e4" candidate_uuid: "09f2de53e052456b8f7eefaed87eed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b076851d96b4de7bb94655a09b884cb" is_pre_election: true
I20260812 11:11:06.191569 25344 raft_consensus.cc:2454] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f2de53e052456b8f7eefaed87eed5f in term 0.
I20260812 11:11:06.191756 25593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c246772f3f541afa922da23885527e4" candidate_uuid: "09f2de53e052456b8f7eefaed87eed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1be81569b3c40b18febbe806036b018" is_pre_election: true
I20260812 11:11:06.192027 25593 raft_consensus.cc:2454] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f2de53e052456b8f7eefaed87eed5f in term 0.
I20260812 11:11:06.192075 25735 leader_election.cc:302] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2de53e052456b8f7eefaed87eed5f, 4b076851d96b4de7bb94655a09b884cb; no voters:
I20260812 11:11:06.192332 25992 raft_consensus.cc:2781] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:06.192471 25992 raft_consensus.cc:482] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:06.192533 25992 raft_consensus.cc:3037] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:06.193575 25992 raft_consensus.cc:504] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.193898 25992 leader_election.cc:288] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [CANDIDATE]: Term 1 election: Requested vote from peers 4b076851d96b4de7bb94655a09b884cb (127.23.97.193:41819), a1be81569b3c40b18febbe806036b018 (127.23.97.194:43875)
I20260812 11:11:06.194252 25344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c246772f3f541afa922da23885527e4" candidate_uuid: "09f2de53e052456b8f7eefaed87eed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b076851d96b4de7bb94655a09b884cb"
I20260812 11:11:06.194355 25593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c246772f3f541afa922da23885527e4" candidate_uuid: "09f2de53e052456b8f7eefaed87eed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1be81569b3c40b18febbe806036b018"
I20260812 11:11:06.194358 25344 raft_consensus.cc:3037] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:06.194482 25593 raft_consensus.cc:3037] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:06.195447 25344 raft_consensus.cc:2454] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2de53e052456b8f7eefaed87eed5f in term 1.
I20260812 11:11:06.195593 25593 raft_consensus.cc:2454] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2de53e052456b8f7eefaed87eed5f in term 1.
I20260812 11:11:06.195744 25735 leader_election.cc:302] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2de53e052456b8f7eefaed87eed5f, 4b076851d96b4de7bb94655a09b884cb; no voters:
I20260812 11:11:06.195891 25992 raft_consensus.cc:2781] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:06.196089 25992 raft_consensus.cc:686] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [term 1 LEADER]: Becoming Leader. State: Replica: 09f2de53e052456b8f7eefaed87eed5f, State: Running, Role: LEADER
I20260812 11:11:06.196458 25992 consensus_queue.cc:237] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } }
I20260812 11:11:06.199656 25056 catalog_manager.cc:5495] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f2de53e052456b8f7eefaed87eed5f (127.23.97.195). New cstate: current_term: 1 leader_uuid: "09f2de53e052456b8f7eefaed87eed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f2de53e052456b8f7eefaed87eed5f" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b076851d96b4de7bb94655a09b884cb" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1be81569b3c40b18febbe806036b018" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43875 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
W20260812 11:11:06.222447 25950 tablet.cc:2369] T 8c246772f3f541afa922da23885527e4 P 09f2de53e052456b8f7eefaed87eed5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:06.225188 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.216324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.23.97.193:41819)
0812 11:11:06.216477 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.216480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.216634 (+ 154us) server_negotiation.cc:407] Connection header received
0812 11:11:06.216722 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.216725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.216763 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.216830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:06.217493 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.218021 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.218910 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.219131 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.219254 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.221852 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:06.221863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:06.221876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:06.221900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:06.223642 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.224046 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.224051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.224052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.224131 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.224427 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.224430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.224431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.224666 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:06.224815 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.224981 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.225073 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260812 11:11:06.246394 25688 tablet.cc:2369] T 8c246772f3f541afa922da23885527e4 P a1be81569b3c40b18febbe806036b018: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:06.254648 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.245010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.23.97.193:41819)
0812 11:11:06.245156 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.245163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.245324 (+ 161us) server_negotiation.cc:407] Connection header received
0812 11:11:06.245445 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.245447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.245487 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.245554 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:06.246353 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.247074 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.248211 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.248492 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.248570 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.251324 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:06.251327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:06.251329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:06.251352 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:06.252845 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.253359 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.253369 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.253371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.253456 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.253746 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.253749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.253751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.253923 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:06.254062 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.254349 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.254474 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260812 11:11:06.267225 25430 tablet.cc:2369] T 8c246772f3f541afa922da23885527e4 P 4b076851d96b4de7bb94655a09b884cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:06.281095 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:06.272830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.23.97.193:41819)
0812 11:11:06.272967 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:06.272971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:06.273185 (+ 214us) server_negotiation.cc:407] Connection header received
0812 11:11:06.273289 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:06.273291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:06.273329 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:06.273395 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:06.274020 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.274488 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.275324 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:06.275522 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:06.275596 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:06.278158 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:06.278161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:06.278163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:06.278186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:06.279392 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.279777 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.279779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.279780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.279850 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:06.280179 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:06.280182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:06.280184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:06.280360 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:06.280530 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:06.280792 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 11:11:06.280885 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 11:11:06.282408 25273 tablet_service.cc:2908] Rejecting scan request for tablet 8c246772f3f541afa922da23885527e4: Uninitialized: clean time has not yet been initialized
I20260812 11:11:06.284479 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25182
I20260812 11:11:06.306802 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25435
I20260812 11:11:06.316684 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25694
I20260812 11:11:06.328550 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24997
2026-08-12T11:11:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2693 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[26020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[26020](info): set up 2 sockets
Aug 12 11:11:06 dist-test-slave-zpvx krb5kdc[26020](info): commencing operation
krb5kdc: starting...
W20260812 11:11:07.629522 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:07 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:07.759279 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:38425
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:38425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:07.872044 26046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:07.872294 26046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:07.872334 26046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:07.876242 26046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:07.876294 26046 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:11:07.876312 26046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:07.876329 26046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:07.876406 26046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:07.876434 26046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:07.881582 26046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:38425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:38425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:07.882579 26046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:07.883605 26046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:07.891942 26061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:07.891928 26053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:07.891933 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:07.892485 26046 server_base.cc:1034] running on GCE node
I20260812 11:11:07.893006 26046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:07.894022 26046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:07.895179 26046 hybrid_clock.cc:648] HybridClock initialized: now 1786533067895149 us; error 44 us; skew 500 ppm
Aug 12 11:11:07 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:07.903378 26046 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:07.904234 26046 webserver.cc:460] Webserver started at http://127.23.97.254:40079/ using document root <none> and password file <none>
I20260812 11:11:07.904824 26046 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:07.904886 26046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:07.905048 26046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:07.907084 26046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "babd588ff2a441978588dde63ebd2c14"
format_stamp: "Formatted at 2026-08-12 11:11:07 on dist-test-slave-zpvx"
server_key: "1c709159bc05ad50c85cb3fc442e3f26"
server_key_iv: "11dddfddc849932dc11f90e4a4dadd6a"
server_key_version: "encryptionkey@0"
I20260812 11:11:07.907605 26046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "babd588ff2a441978588dde63ebd2c14"
format_stamp: "Formatted at 2026-08-12 11:11:07 on dist-test-slave-zpvx"
server_key: "1c709159bc05ad50c85cb3fc442e3f26"
server_key_iv: "11dddfddc849932dc11f90e4a4dadd6a"
server_key_version: "encryptionkey@0"
I20260812 11:11:07.911288 26046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:11:07.913743 26069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:07.915030 26046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:07.915135 26046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "babd588ff2a441978588dde63ebd2c14"
format_stamp: "Formatted at 2026-08-12 11:11:07 on dist-test-slave-zpvx"
server_key: "1c709159bc05ad50c85cb3fc442e3f26"
server_key_iv: "11dddfddc849932dc11f90e4a4dadd6a"
server_key_version: "encryptionkey@0"
I20260812 11:11:07.915263 26046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:07.933795 26046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:07.937146 26046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:07.937319 26046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:07.945457 26046 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:38425
I20260812 11:11:07.945487 26179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:38425 every 8 connection(s)
I20260812 11:11:07.946470 26046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:07.949290 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:07.954813 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Bootstrap starting.
I20260812 11:11:07.956001 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26046
I20260812 11:11:07.956095 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:07.956346 23943 external_mini_cluster.cc:1428] Setting key 365abb73962f877ae27699d66e04150c
I20260812 11:11:07.957394 26180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:07.958237 26180 log.cc:824] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:07.960266 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: No bootstrap required, opened a new log
I20260812 11:11:07.963869 26180 raft_consensus.cc:348] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } }
I20260812 11:11:07.964118 26180 raft_consensus.cc:374] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:07.964200 26180 raft_consensus.cc:729] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babd588ff2a441978588dde63ebd2c14, State: Initialized, Role: FOLLOWER
I20260812 11:11:07.964723 26180 consensus_queue.cc:260] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } }
I20260812 11:11:07.964879 26180 raft_consensus.cc:388] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:07.964951 26180 raft_consensus.cc:482] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:07.965054 26180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 11:11:07 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:07.965991 26180 raft_consensus.cc:504] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } }
I20260812 11:11:07.966334 26180 leader_election.cc:302] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: babd588ff2a441978588dde63ebd2c14; no voters:
I20260812 11:11:07.966703 26180 leader_election.cc:288] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:07.966838 26189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:07.967067 26189 raft_consensus.cc:686] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [term 1 LEADER]: Becoming Leader. State: Replica: babd588ff2a441978588dde63ebd2c14, State: Running, Role: LEADER
I20260812 11:11:07.967404 26189 consensus_queue.cc:237] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } }
I20260812 11:11:07.967988 26180 sys_catalog.cc:564] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:07.968693 26189 sys_catalog.cc:455] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader babd588ff2a441978588dde63ebd2c14. Latest consensus state: current_term: 1 leader_uuid: "babd588ff2a441978588dde63ebd2c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } } }
I20260812 11:11:07.968818 26189 sys_catalog.cc:458] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:07.969223 26199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:07.970844 26190 sys_catalog.cc:455] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "babd588ff2a441978588dde63ebd2c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babd588ff2a441978588dde63ebd2c14" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38425 } } }
I20260812 11:11:07.970974 26190 sys_catalog.cc:458] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:07.972093 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:07.957481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.23.97.254:38425)
0812 11:11:07.957938 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:07.957953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:07.957988 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:07.958686 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:07.958710 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:07.959125 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:07.959469 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:07.960344 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:07.961735 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:07.962624 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:07.963133 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:07.963312 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:07.966080 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:07.966092 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:07.966106 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:07.966157 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:07.968296 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:07.968996 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:07.969005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:07.969014 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:07.969212 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:07.969608 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:07.969612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:07.969613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:07.970210 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:07.970417 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:07.970897 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 11:11:07.971184 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":150,"threads_started":1}
I20260812 11:11:07.972834 26199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:07.978731 26199 catalog_manager.cc:1347] Generated new cluster ID: 2493d8d40671476483fd42a9e1cc8da8
I20260812 11:11:07.978794 26199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:07.983620 26199 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:07.984800 26199 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:07.994620 26199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Generated new TSK 0
I20260812 11:11:07.995275 26199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:08.031033 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38425
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:08.144330 26222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:08.144632 26222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:08.144708 26222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:08.148692 26222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:08.148767 26222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:08.148835 26222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:08.148911 26222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:08.154024 26222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:08.155104 26222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:08.156136 26222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:08.165660 26238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:08.165671 26236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:08.165742 26232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:08.165822 26222 server_base.cc:1034] running on GCE node
I20260812 11:11:08.166359 26222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:08.167044 26222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:08.168263 26222 hybrid_clock.cc:648] HybridClock initialized: now 1786533068168219 us; error 50 us; skew 500 ppm
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:08.171311 26222 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:08.172086 26222 webserver.cc:460] Webserver started at http://127.23.97.193:33607/ using document root <none> and password file <none>
I20260812 11:11:08.172657 26222 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:08.172726 26222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:08.172889 26222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:08.174819 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "81c0cce09e364e2294758936e8966aed"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "68420f6719c6eccc9c109a9a226c8fab"
server_key_iv: "be7388c9ab37f05f2e4cc1fdf0d00bc9"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.175339 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "81c0cce09e364e2294758936e8966aed"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "68420f6719c6eccc9c109a9a226c8fab"
server_key_iv: "be7388c9ab37f05f2e4cc1fdf0d00bc9"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.178948 26222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 11:11:08.181373 26248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.182567 26222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:08.182672 26222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "81c0cce09e364e2294758936e8966aed"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "68420f6719c6eccc9c109a9a226c8fab"
server_key_iv: "be7388c9ab37f05f2e4cc1fdf0d00bc9"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.182756 26222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:08.193602 26222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:08.196183 26222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:08.196336 26222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:08.197783 26222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:08.197860 26222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.197932 26222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:08.197976 26222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:08.209425 26222 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44703
I20260812 11:11:08.210207 26459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44703 every 8 connection(s)
I20260812 11:11:08.210464 26222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:08.212242 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.198568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:39269 (local address 127.23.97.254:38425)
0812 11:11:08.198704 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.198707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.200329 (+ 1622us) server_negotiation.cc:407] Connection header received
0812 11:11:08.201494 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.201497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.201558 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.201665 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.202928 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.203500 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.204771 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.205075 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.205202 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.208052 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.208057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.208060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.208091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.209971 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.210605 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.210613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.210614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.210704 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.211072 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.211077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.211078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.211277 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.211392 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.211930 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.212066 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:11:08.212963 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.199629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.193:39269)
0812 11:11:08.200171 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.200222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.201283 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.201856 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.201863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.202262 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.202776 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.202790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.203635 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.203638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.204620 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.204655 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.205246 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.205250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.205425 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.206043 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.206069 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.207885 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.210101 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.210107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.210121 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.210463 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.210802 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.210807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.210809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.210980 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.211391 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.211398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.211690 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.211885 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.212198 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
I20260812 11:11:08.214412 26461 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38425
I20260812 11:11:08.214569 26461 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:08.215050 26461 heartbeater.cc:511] Master 127.23.97.254:38425 requested a full tablet report, sending...
I20260812 11:11:08.216646 26106 ts_manager.cc:194] Registered new tserver with Master: 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703)
I20260812 11:11:08.217811 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26222
I20260812 11:11:08.217947 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:08.218185 23943 external_mini_cluster.cc:1428] Setting key 4268254d33ecc6e6b63ab0b00846a581
I20260812 11:11:08.218192 26106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:39269
I20260812 11:11:08.224543 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.216600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.193:50111)
0812 11:11:08.216957 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.216971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.217069 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.217294 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.217297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.217500 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.217639 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.217648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.218984 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.218987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.219560 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.219594 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.220124 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.220126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.220224 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.220758 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.220769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.221064 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.222938 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.222940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.222942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.223150 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.223463 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.223466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.223467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.223560 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.224011 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.224013 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.224116 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.224205 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.224305 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":8,"thread_start_us":116,"threads_started":1}
I20260812 11:11:08.224691 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.216655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:50111 (local address 127.23.97.254:38425)
0812 11:11:08.216781 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.216785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.217047 (+ 262us) server_negotiation.cc:407] Connection header received
0812 11:11:08.217148 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.217150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.217188 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.217252 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.217785 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.218532 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.219707 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.220014 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.220122 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.221206 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.221209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.221212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.221246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.222801 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.223257 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.223264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.223266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.223360 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.223662 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.223667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.223669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.223902 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.224010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.224298 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.224467 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:08.249658 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38425
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:08.363255 26479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:08.363495 26479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:08.363539 26479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:08.367578 26479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:08.367632 26479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:08.367671 26479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:08.367709 26479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:08.373064 26479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:08.373999 26479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:08.374967 26479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:08.384927 26495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:08.384904 26493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:08.384905 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:08.385345 26479 server_base.cc:1034] running on GCE node
I20260812 11:11:08.385753 26479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:08.386425 26479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:08.387633 26479 hybrid_clock.cc:648] HybridClock initialized: now 1786533068387617 us; error 33 us; skew 500 ppm
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:08.390457 26479 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:08.391347 26479 webserver.cc:460] Webserver started at http://127.23.97.194:35515/ using document root <none> and password file <none>
I20260812 11:11:08.391896 26479 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:08.391989 26479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:08.392225 26479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:08.394270 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "6f3ab6e2086d46df992a99894606e735"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "aa43ad61ff2e4bbc04c94ba29e971e30"
server_key_iv: "651aa680dc4142c5e74cff544694c88a"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.394843 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "6f3ab6e2086d46df992a99894606e735"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "aa43ad61ff2e4bbc04c94ba29e971e30"
server_key_iv: "651aa680dc4142c5e74cff544694c88a"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.398633 26479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:11:08.401237 26518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.402529 26479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:11:08.402685 26479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "6f3ab6e2086d46df992a99894606e735"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "aa43ad61ff2e4bbc04c94ba29e971e30"
server_key_iv: "651aa680dc4142c5e74cff544694c88a"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.402802 26479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:08.412956 26479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:08.415889 26479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:08.416083 26479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:08.417642 26479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:08.417727 26479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.417796 26479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:08.417836 26479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:08.429019 26479 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:32781
I20260812 11:11:08.429173 26722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:32781 every 8 connection(s)
I20260812 11:11:08.430162 26479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:08.432754 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.418304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:57697 (local address 127.23.97.254:38425)
0812 11:11:08.418467 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.418472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.419927 (+ 1455us) server_negotiation.cc:407] Connection header received
0812 11:11:08.421417 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.421424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.421495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.421565 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.423195 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.423733 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.425552 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.425782 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.425862 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.429085 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.429091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.429096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.429126 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.430574 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.431179 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.431183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.431184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.431277 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.431616 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.431619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.431621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.431817 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.431925 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.432459 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.432584 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:11:08.433547 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.419291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.194:57697)
0812 11:11:08.419764 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.419822 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.421139 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.421786 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.421798 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.422319 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.423051 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.423075 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.423844 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.423848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.425430 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.425441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.426038 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.426043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.426343 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.427010 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.427034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.428898 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.430732 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.430737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.430750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.431063 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.431371 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.431374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.431376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.431509 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.431925 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.431932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.432180 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.432399 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.432701 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260812 11:11:08.434752 26723 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38425
I20260812 11:11:08.434916 26723 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:08.435468 26723 heartbeater.cc:511] Master 127.23.97.254:38425 requested a full tablet report, sending...
I20260812 11:11:08.436403 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26479
I20260812 11:11:08.436523 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:08.436517 26107 ts_manager.cc:194] Registered new tserver with Master: 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781)
I20260812 11:11:08.436861 23943 external_mini_cluster.cc:1428] Setting key 8069874bd50461962ee36188b4bd341a
I20260812 11:11:08.437301 26107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:57697
I20260812 11:11:08.444439 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.436872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.194:44499)
0812 11:11:08.437177 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.437192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.437288 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.437880 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.437883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.438097 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.438207 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.438214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.439235 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.439238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.439894 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.439904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.440397 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.440404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.440505 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.440977 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.440988 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.441273 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.442838 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.442841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.442843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.443072 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.443422 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.443424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.443426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.443500 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.444013 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.444015 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.444084 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.444164 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.444264 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":126,"threads_started":1}
I20260812 11:11:08.444639 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.437583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:44499 (local address 127.23.97.254:38425)
0812 11:11:08.437685 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.437689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.437702 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:08.437737 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.437739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.437770 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.437860 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.438345 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.439070 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.440027 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.440256 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.440360 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.441417 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.441420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.441422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.441446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.442717 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.443196 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.443203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.443208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.443316 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.443594 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.443597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.443599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.443907 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.444028 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.444256 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.444420 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:08.465608 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38425
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:08.577262 26735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:08.577524 26735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:08.577601 26735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:08.581712 26735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:08.581795 26735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:08.581875 26735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:08.581959 26735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:08.587072 26735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:08.588035 26735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:08.589038 26735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:08.598300 26749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:08.598318 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:08.598374 26735 server_base.cc:1034] running on GCE node
W20260812 11:11:08.598353 26744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:08.598868 26735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:08.599488 26735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:08.600675 26735 hybrid_clock.cc:648] HybridClock initialized: now 1786533068600664 us; error 29 us; skew 500 ppm
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:08.603379 26735 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:08.604190 26735 webserver.cc:460] Webserver started at http://127.23.97.195:33765/ using document root <none> and password file <none>
I20260812 11:11:08.604796 26735 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:08.604890 26735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:08.605121 26735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:08.607244 26735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "92d8b6ba40d64690ae2d5fd34f5252c1"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "750875176dc2d9f2c456e62955c06301"
server_key_iv: "b68ab020137a006b8d6bf87f1be624e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.607865 26735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "92d8b6ba40d64690ae2d5fd34f5252c1"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "750875176dc2d9f2c456e62955c06301"
server_key_iv: "b68ab020137a006b8d6bf87f1be624e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.611452 26735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:11:08.613837 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.615031 26735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:08.615213 26735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "92d8b6ba40d64690ae2d5fd34f5252c1"
format_stamp: "Formatted at 2026-08-12 11:11:08 on dist-test-slave-zpvx"
server_key: "750875176dc2d9f2c456e62955c06301"
server_key_iv: "b68ab020137a006b8d6bf87f1be624e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:08.615325 26735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:08.638777 26735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:08.642146 26735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:08.642360 26735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:08.643821 26735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:08.643894 26735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:08.643970 26735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:08.644022 26735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:08.655151 26735 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:43451
I20260812 11:11:08.655292 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:43451 every 8 connection(s)
I20260812 11:11:08.656080 26735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:08.659053 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.644735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:39621 (local address 127.23.97.254:38425)
0812 11:11:08.644881 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.644884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.645934 (+ 1050us) server_negotiation.cc:407] Connection header received
0812 11:11:08.646886 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.646889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.646952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.647058 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.648692 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.649626 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.651488 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.651905 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.652034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.654976 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.654980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.654983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.655017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.656821 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.657482 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.657485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.657487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.657584 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.658058 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.658061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.658062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.658282 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.658384 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.658830 (+ 446us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.658934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:11:08.659868 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.645293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.195:39621)
0812 11:11:08.645785 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.645827 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.646669 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.647240 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.647250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.647759 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.648504 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.648524 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.649793 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.649800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.651317 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.651330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.652055 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.652058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.652191 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.652785 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.652805 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.654736 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.656947 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.656956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.656989 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.657372 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.657713 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.657726 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.657729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.657952 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.658392 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.658397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.658608 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.658767 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.659002 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260812 11:11:08.661038 26964 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38425
I20260812 11:11:08.661201 26964 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:08.661697 26964 heartbeater.cc:511] Master 127.23.97.254:38425 requested a full tablet report, sending...
I20260812 11:11:08.661820 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26735
I20260812 11:11:08.661959 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:08.662279 23943 external_mini_cluster.cc:1428] Setting key 5f225f3d47e8f3d8ee7ccc037fea492b
I20260812 11:11:08.662822 26106 ts_manager.cc:194] Registered new tserver with Master: 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195:43451)
I20260812 11:11:08.663546 26106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:39621
I20260812 11:11:08.664449 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:11:08.671446 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.663527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.23.97.195:40193)
0812 11:11:08.663843 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.663859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.663938 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.664540 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.664543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.664747 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:08.664872 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.664879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.665872 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.665875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.666399 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.666406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.667009 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.667015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.667132 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.667596 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:08.667615 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:08.668050 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:08.669939 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.669945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.669951 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.670253 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.670526 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:08.670531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:08.670534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:08.670639 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:08.671011 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:08.671017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:08.671126 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.671218 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.671327 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260812 11:11:08.671947 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.663618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:40193 (local address 127.23.97.254:38425)
0812 11:11:08.664241 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.664245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.664258 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:08.664291 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.664293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.664333 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.664434 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.665000 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.665740 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.666511 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.666876 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.666968 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.668176 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.668179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.668181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.668202 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.669836 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.670356 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.670359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.670361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.670432 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.670738 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.670740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.670741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.670890 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.671006 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.671325 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.671455 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:11:08 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:08.695411 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.684193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.23.97.254:38425)
0812 11:11:08.684316 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.684319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.684330 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:08.684360 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.684362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.684450 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.684514 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:08.685289 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.686199 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.687057 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.687456 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.687573 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.691344 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:08.691347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:08.691350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:08.691374 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:08.693504 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.694134 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.694139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.694144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.694235 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:08.694569 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:08.694572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:08.694574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:08.694740 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:08.694899 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.695131 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.695253 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:11:08.698027 26107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:08.701020 26107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:08.715157 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.709035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:32781 (local address 127.0.0.1:55760)
0812 11:11:08.710049 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.710065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.710155 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.710648 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.710653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.710675 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:08.710812 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.710820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.712244 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.712248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.713535 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.713546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.714555 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.714559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.714680 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.714740 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.714873 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.714988 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":109,"thread_start_us":91,"threads_started":1}
I20260812 11:11:08.715586 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.709804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.23.97.195:43451)
0812 11:11:08.710958 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.710967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.710984 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:11:08.711039 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.711043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.711209 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.711365 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:08.711694 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.712712 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.713948 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.714771 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.714870 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.715020 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.715116 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.715199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":118,"threads_started":1}
I20260812 11:11:08.715843 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.709149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.23.97.194:32781)
0812 11:11:08.710248 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.710258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.710283 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:11:08.710346 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.710350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.710497 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.710661 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:08.710942 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.712110 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.713666 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.714427 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.714544 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.714714 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.715581 (+ 867us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.715676 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":109,"threads_started":1}
I20260812 11:11:08.716012 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.709136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43451 (local address 127.0.0.1:51252)
0812 11:11:08.709878 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.709933 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.710054 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.711514 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.711516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.711530 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:08.711586 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.711589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.712830 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.712833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.713818 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.713829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.714885 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.714888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.714981 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.715001 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.715086 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.715139 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":117,"threads_started":1}
I20260812 11:11:08.716992 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.709821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44703 (local address 127.0.0.1:53954)
0812 11:11:08.710614 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.710628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.710751 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.711232 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.711235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.711252 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:08.711368 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.711374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.713275 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.713279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.714574 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.714585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.715864 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.715869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.716699 (+ 830us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.716723 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.716813 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.716880 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":88,"threads_started":1}
I20260812 11:11:08.717485 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.710251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.23.97.193:44703)
0812 11:11:08.710528 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.710545 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.710721 (+ 176us) server_negotiation.cc:407] Connection header received
0812 11:11:08.710867 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.710871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.711086 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.711253 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:08.711479 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.713155 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.714719 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.715535 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.715644 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.715795 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.717268 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.717365 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":109,"threads_started":1}
I20260812 11:11:08.718683 26608 tablet_service.cc:1467] Processing CreateTablet for tablet 02d16b41115444b194812a61ac7cb712 (DEFAULT_TABLE table=test-table [id=ae81d639df874e7a942ad5b2ba79bca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.718693 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 02d16b41115444b194812a61ac7cb712 (DEFAULT_TABLE table=test-table [id=ae81d639df874e7a942ad5b2ba79bca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.719746 26608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d16b41115444b194812a61ac7cb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:08.719800 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d16b41115444b194812a61ac7cb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:08.719918 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 02d16b41115444b194812a61ac7cb712 (DEFAULT_TABLE table=test-table [id=ae81d639df874e7a942ad5b2ba79bca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.720826 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d16b41115444b194812a61ac7cb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:08.726666 27009 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Bootstrap starting.
I20260812 11:11:08.727952 27011 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Bootstrap starting.
I20260812 11:11:08.728686 27010 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Bootstrap starting.
I20260812 11:11:08.729140 27009 tablet_bootstrap.cc:654] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:08.730103 27011 tablet_bootstrap.cc:654] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:08.730168 27009 log.cc:824] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:08.730935 27010 tablet_bootstrap.cc:654] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:08.731464 27011 log.cc:824] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:08.731626 27010 log.cc:824] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:08.732462 27009 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: No bootstrap required, opened a new log
I20260812 11:11:08.732688 27009 ts_tablet_manager.cc:1397] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 11:11:08.733358 27011 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: No bootstrap required, opened a new log
I20260812 11:11:08.733474 27010 tablet_bootstrap.cc:492] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: No bootstrap required, opened a new log
I20260812 11:11:08.733542 27011 ts_tablet_manager.cc:1397] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:08.733621 27010 ts_tablet_manager.cc:1397] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:11:08.736279 27009 raft_consensus.cc:348] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.736577 27009 raft_consensus.cc:374] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:08.736636 27009 raft_consensus.cc:729] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3ab6e2086d46df992a99894606e735, State: Initialized, Role: FOLLOWER
I20260812 11:11:08.737061 27009 consensus_queue.cc:260] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.737365 27010 raft_consensus.cc:348] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.737365 27011 raft_consensus.cc:348] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.737622 27011 raft_consensus.cc:374] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:08.737622 27010 raft_consensus.cc:374] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:08.737691 27011 raft_consensus.cc:729] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c0cce09e364e2294758936e8966aed, State: Initialized, Role: FOLLOWER
I20260812 11:11:08.737680 27010 raft_consensus.cc:729] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d8b6ba40d64690ae2d5fd34f5252c1, State: Initialized, Role: FOLLOWER
I20260812 11:11:08.737772 26723 heartbeater.cc:503] Master 127.23.97.254:38425 was elected leader, sending a full tablet report...
I20260812 11:11:08.738075 27009 ts_tablet_manager.cc:1428] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 11:11:08.738104 27010 consensus_queue.cc:260] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.738121 27011 consensus_queue.cc:260] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.738945 26461 heartbeater.cc:503] Master 127.23.97.254:38425 was elected leader, sending a full tablet report...
I20260812 11:11:08.738945 26964 heartbeater.cc:503] Master 127.23.97.254:38425 was elected leader, sending a full tablet report...
I20260812 11:11:08.739024 27011 ts_tablet_manager.cc:1428] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:08.739024 27010 ts_tablet_manager.cc:1428] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260812 11:11:08.907147 26967 tablet.cc:2369] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:08.909550 27022 raft_consensus.cc:482] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:08.909732 27022 raft_consensus.cc:504] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.910933 27022 leader_election.cc:288] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781), 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703)
I20260812 11:11:08.915572 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.911224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:32781 (local address 127.23.97.195:49849)
0812 11:11:08.911390 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.911408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.911522 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.911977 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.911983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.912007 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:08.912139 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.912149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.913257 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.913261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.914323 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.914333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.915194 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.915197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.915280 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.915301 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.915363 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.915416 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 11:11:08.915639 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.911295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49849 (local address 127.23.97.194:32781)
0812 11:11:08.911441 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.911448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.911467 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:08.911678 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.911681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.911751 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.911822 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:08.912274 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.913153 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.914448 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.915089 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.915206 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.915256 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.915409 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.915504 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:08.916281 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d16b41115444b194812a61ac7cb712" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735" is_pre_election: true
I20260812 11:11:08.916709 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.911678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:55553 (local address 127.23.97.193:44703)
0812 11:11:08.911829 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.911833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.911955 (+ 122us) server_negotiation.cc:407] Connection header received
0812 11:11:08.912035 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.912038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.912085 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.912172 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:08.913088 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.913881 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.915524 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.916100 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.916226 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.916275 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.916505 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.916596 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:11:08.916688 26642 raft_consensus.cc:2454] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 0.
I20260812 11:11:08.917073 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.911526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44703 (local address 127.23.97.195:55553)
0812 11:11:08.911859 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:08.911871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:08.911957 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:08.912185 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:08.912187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:08.912220 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:08.912953 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.912964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.913999 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.914001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.915389 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:08.915399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.916231 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.916234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:08.916309 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.916326 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:11:08.916886 (+ 560us) client_negotiation.cc:240] Negotiation successful
0812 11:11:08.916961 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":163,"threads_started":1}
I20260812 11:11:08.917240 26760 leader_election.cc:302] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3ab6e2086d46df992a99894606e735, 92d8b6ba40d64690ae2d5fd34f5252c1; no voters:
I20260812 11:11:08.917547 27022 raft_consensus.cc:2781] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:08.917639 27022 raft_consensus.cc:482] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:08.917697 27022 raft_consensus.cc:3037] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:08.917971 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d16b41115444b194812a61ac7cb712" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c0cce09e364e2294758936e8966aed" is_pre_election: true
I20260812 11:11:08.918231 26375 raft_consensus.cc:2454] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 0.
I20260812 11:11:08.919044 27022 raft_consensus.cc:504] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.919415 27022 leader_election.cc:288] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781), 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703)
I20260812 11:11:08.919736 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d16b41115444b194812a61ac7cb712" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735"
I20260812 11:11:08.919876 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d16b41115444b194812a61ac7cb712" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c0cce09e364e2294758936e8966aed"
I20260812 11:11:08.919881 26642 raft_consensus.cc:3037] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:08.919984 26375 raft_consensus.cc:3037] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:08.921222 26375 raft_consensus.cc:2454] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 1.
I20260812 11:11:08.921228 26642 raft_consensus.cc:2454] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 1.
I20260812 11:11:08.921521 26760 leader_election.cc:302] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3ab6e2086d46df992a99894606e735, 92d8b6ba40d64690ae2d5fd34f5252c1; no voters:
I20260812 11:11:08.921690 27022 raft_consensus.cc:2781] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:08.921913 27022 raft_consensus.cc:686] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 LEADER]: Becoming Leader. State: Replica: 92d8b6ba40d64690ae2d5fd34f5252c1, State: Running, Role: LEADER
I20260812 11:11:08.922250 27022 consensus_queue.cc:237] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } }
I20260812 11:11:08.925486 26106 catalog_manager.cc:5495] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } health_report { overall_health: UNKNOWN } } }
W20260812 11:11:08.931331 26724 tablet.cc:2369] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:08.960032 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:08.955969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.23.97.195:43451)
0812 11:11:08.956127 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:08.956132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:08.956154 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:11:08.956427 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:08.956430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:08.956496 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:08.956589 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:08.956849 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.957629 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.958479 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:08.958825 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:08.958922 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:08.959201 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:08.959654 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:08.959747 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:08.959815 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 11:11:08.959866 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 11:11:08.962194 26463 tablet.cc:2369] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:08.965600 26375 raft_consensus.cc:1261] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Refusing update from remote peer 92d8b6ba40d64690ae2d5fd34f5252c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:08.965620 26642 raft_consensus.cc:1261] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Refusing update from remote peer 92d8b6ba40d64690ae2d5fd34f5252c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:08.966239 27022 consensus_queue.cc:1035] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:08.966416 27024 consensus_queue.cc:1035] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:08.973392 27029 mvcc.cc:205] Tried to move back new op lower bound from 7317639450476625920 to 7317639450308190208. Current Snapshot: MvccSnapshot[applied={T|T < 7317639450476625920}]
I20260812 11:11:08.974861 27032 mvcc.cc:205] Tried to move back new op lower bound from 7317639450476625920 to 7317639450308190208. Current Snapshot: MvccSnapshot[applied={T|T < 7317639450476625920}]
I20260812 11:11:08.985232 26106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:08.985452 26106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:08.988174 26106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Sending DeleteTablet for 3 replicas of tablet 02d16b41115444b194812a61ac7cb712
I20260812 11:11:08.989122 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d16b41115444b194812a61ac7cb712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:08 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:51252
I20260812 11:11:08.989109 26608 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d16b41115444b194812a61ac7cb712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:08 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:55760
I20260812 11:11:08.989750 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d16b41115444b194812a61ac7cb712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:08 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260812 11:11:08.989817 27041 tablet_replica.cc:330] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: stopping tablet replica
I20260812 11:11:08.990034 26106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:11:08.990149 27041 raft_consensus.cc:2229] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 11:11:08.990415 26106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:08.990430 27041 raft_consensus.cc:2258] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:08.990805 27046 tablet_replica.cc:330] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: stopping tablet replica
I20260812 11:11:08.991083 27046 raft_consensus.cc:2229] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:08.991420 27046 raft_consensus.cc:2258] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:08.991976 27041 ts_tablet_manager.cc:1905] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:08.992846 27046 ts_tablet_manager.cc:1905] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:08.994839 27041 ts_tablet_manager.cc:1918] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:08.994946 27041 log.cc:1196] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/02d16b41115444b194812a61ac7cb712
I20260812 11:11:08.995276 27041 ts_tablet_manager.cc:1939] T 02d16b41115444b194812a61ac7cb712 P 6f3ab6e2086d46df992a99894606e735: Deleting consensus metadata
I20260812 11:11:08.995512 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e (DEFAULT_TABLE table=test-table [id=dd1648f3548e47178e22514902ddf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.995697 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e (DEFAULT_TABLE table=test-table [id=dd1648f3548e47178e22514902ddf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.995779 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572eeab8fd9f4e4d8e833f141f85579e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:08.996141 26608 tablet_service.cc:1467] Processing CreateTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e (DEFAULT_TABLE table=test-table [id=dd1648f3548e47178e22514902ddf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:08.996562 26072 catalog_manager.cc:4841] TS 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781): tablet 02d16b41115444b194812a61ac7cb712 (table test-table [id=ae81d639df874e7a942ad5b2ba79bca1]) successfully deleted
I20260812 11:11:08.996768 26608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572eeab8fd9f4e4d8e833f141f85579e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:08.998122 27010 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Bootstrap starting.
I20260812 11:11:08.998260 27046 ts_tablet_manager.cc:1918] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:08.998361 27046 log.cc:1196] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/02d16b41115444b194812a61ac7cb712
I20260812 11:11:08.998708 27046 ts_tablet_manager.cc:1939] T 02d16b41115444b194812a61ac7cb712 P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting consensus metadata
I20260812 11:11:08.999070 27009 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Bootstrap starting.
I20260812 11:11:08.999116 27010 tablet_bootstrap.cc:654] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:08.999640 27050 tablet_replica.cc:330] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: stopping tablet replica
I20260812 11:11:08.999935 27050 raft_consensus.cc:2229] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:09.000064 27009 tablet_bootstrap.cc:654] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:09.000224 27050 raft_consensus.cc:2258] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:09.001331 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572eeab8fd9f4e4d8e833f141f85579e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:09.001672 26074 catalog_manager.cc:4841] TS 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195:43451): tablet 02d16b41115444b194812a61ac7cb712 (table test-table [id=ae81d639df874e7a942ad5b2ba79bca1]) successfully deleted
I20260812 11:11:09.002281 27050 ts_tablet_manager.cc:1905] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:09.003409 27011 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Bootstrap starting.
I20260812 11:11:09.004310 27009 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: No bootstrap required, opened a new log
I20260812 11:11:09.004433 27009 ts_tablet_manager.cc:1397] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 11:11:09.004452 27011 tablet_bootstrap.cc:654] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:09.004925 27009 raft_consensus.cc:348] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.005051 27009 raft_consensus.cc:374] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.005090 27009 raft_consensus.cc:729] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3ab6e2086d46df992a99894606e735, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.005239 27009 consensus_queue.cc:260] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.005501 27009 ts_tablet_manager.cc:1428] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:09.005497 27050 ts_tablet_manager.cc:1918] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:09.005597 27050 log.cc:1196] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/02d16b41115444b194812a61ac7cb712
I20260812 11:11:09.006307 27050 ts_tablet_manager.cc:1939] T 02d16b41115444b194812a61ac7cb712 P 81c0cce09e364e2294758936e8966aed: Deleting consensus metadata
I20260812 11:11:09.006830 27010 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: No bootstrap required, opened a new log
I20260812 11:11:09.006907 27010 ts_tablet_manager.cc:1397] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260812 11:11:09.007390 27010 raft_consensus.cc:348] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.007517 27010 raft_consensus.cc:374] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.007555 27010 raft_consensus.cc:729] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d8b6ba40d64690ae2d5fd34f5252c1, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.007678 27010 consensus_queue.cc:260] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.007962 27010 ts_tablet_manager.cc:1428] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:09.008008 26072 catalog_manager.cc:4841] TS 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703): tablet 02d16b41115444b194812a61ac7cb712 (table test-table [id=ae81d639df874e7a942ad5b2ba79bca1]) successfully deleted
I20260812 11:11:09.009210 27011 tablet_bootstrap.cc:492] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: No bootstrap required, opened a new log
I20260812 11:11:09.009298 27011 ts_tablet_manager.cc:1397] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260812 11:11:09.009712 27011 raft_consensus.cc:348] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.009822 27011 raft_consensus.cc:374] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.009855 27011 raft_consensus.cc:729] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c0cce09e364e2294758936e8966aed, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.009969 27011 consensus_queue.cc:260] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.010228 27011 ts_tablet_manager.cc:1428] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:09.191293 27024 raft_consensus.cc:482] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:09.191421 27024 raft_consensus.cc:504] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.191847 27024 leader_election.cc:288] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703), 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781)
I20260812 11:11:09.192320 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572eeab8fd9f4e4d8e833f141f85579e" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c0cce09e364e2294758936e8966aed" is_pre_election: true
I20260812 11:11:09.192472 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572eeab8fd9f4e4d8e833f141f85579e" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735" is_pre_election: true
I20260812 11:11:09.192553 26375 raft_consensus.cc:2454] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 0.
I20260812 11:11:09.192679 26642 raft_consensus.cc:2454] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 0.
I20260812 11:11:09.193048 26760 leader_election.cc:302] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81c0cce09e364e2294758936e8966aed, 92d8b6ba40d64690ae2d5fd34f5252c1; no voters:
I20260812 11:11:09.193234 27024 raft_consensus.cc:2781] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:09.193300 27024 raft_consensus.cc:482] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:09.193327 27024 raft_consensus.cc:3037] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.194415 27024 raft_consensus.cc:504] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.194743 27024 leader_election.cc:288] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 election: Requested vote from peers 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703), 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781)
I20260812 11:11:09.195163 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572eeab8fd9f4e4d8e833f141f85579e" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c0cce09e364e2294758936e8966aed"
I20260812 11:11:09.195322 26375 raft_consensus.cc:3037] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.195325 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572eeab8fd9f4e4d8e833f141f85579e" candidate_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735"
I20260812 11:11:09.195459 26642 raft_consensus.cc:3037] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.196307 26375 raft_consensus.cc:2454] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 1.
I20260812 11:11:09.196661 26760 leader_election.cc:302] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81c0cce09e364e2294758936e8966aed, 92d8b6ba40d64690ae2d5fd34f5252c1; no voters:
I20260812 11:11:09.196803 26642 raft_consensus.cc:2454] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d8b6ba40d64690ae2d5fd34f5252c1 in term 1.
I20260812 11:11:09.196843 27024 raft_consensus.cc:2781] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:09.196959 27024 raft_consensus.cc:686] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 LEADER]: Becoming Leader. State: Replica: 92d8b6ba40d64690ae2d5fd34f5252c1, State: Running, Role: LEADER
I20260812 11:11:09.197149 27024 consensus_queue.cc:237] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.198772 26106 catalog_manager.cc:5495] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } health_report { overall_health: HEALTHY } } }
Aug 12 11:11:09 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533067, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:09.258481 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.250107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.23.97.254:38425)
0812 11:11:09.250363 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.250373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.250458 (+ 85us) server_negotiation.cc:407] Connection header received
0812 11:11:09.250613 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.250616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.250662 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.250754 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:09.251404 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.251947 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.252802 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.253030 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.253139 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.255381 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:09.255388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:09.255390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:09.255410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:09.256764 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.257225 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.257229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.257231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.257314 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.257581 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.257586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.257588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.257756 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:09.257874 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.258162 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.258350 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260812 11:11:09.258481 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.250023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38425 (local address 127.0.0.1:35846)
0812 11:11:09.250349 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.250367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.250520 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.250771 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.250774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.251194 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:09.251286 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.251292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.252057 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.252062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.252683 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.252692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.253140 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.253143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.253224 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.253766 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:09.253779 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:09.255234 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:09.256888 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:09.256897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:09.256904 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:09.257123 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:09.257405 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:09.257410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:09.257412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:09.257492 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:09.257875 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:09.257883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:09.257990 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.258114 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.258277 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260812 11:11:09.262552 26106 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:35846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:11:09.263348 26106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:09.267970 26608 tablet_service.cc:1467] Processing CreateTablet for tablet 972f1f1cf5b34b8f8caef910317bac9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2708a9f3e35b4ccf99400ebae507100c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:09.268078 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 972f1f1cf5b34b8f8caef910317bac9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2708a9f3e35b4ccf99400ebae507100c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:09.268591 26608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972f1f1cf5b34b8f8caef910317bac9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:09.268599 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972f1f1cf5b34b8f8caef910317bac9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:09.267970 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 972f1f1cf5b34b8f8caef910317bac9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2708a9f3e35b4ccf99400ebae507100c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:09.269553 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972f1f1cf5b34b8f8caef910317bac9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:09.271157 27010 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: Bootstrap starting.
I20260812 11:11:09.272040 27009 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: Bootstrap starting.
I20260812 11:11:09.272142 27010 tablet_bootstrap.cc:654] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:09.272490 27011 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: Bootstrap starting.
I20260812 11:11:09.273072 27009 tablet_bootstrap.cc:654] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:09.273389 27010 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: No bootstrap required, opened a new log
I20260812 11:11:09.273490 27010 ts_tablet_manager.cc:1397] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 11:11:09.273579 27011 tablet_bootstrap.cc:654] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:09.273911 27010 raft_consensus.cc:348] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.274008 27010 raft_consensus.cc:374] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.274030 27010 raft_consensus.cc:729] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d8b6ba40d64690ae2d5fd34f5252c1, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.274179 27010 consensus_queue.cc:260] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.274559 27010 ts_tablet_manager.cc:1428] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:09.274854 27011 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: No bootstrap required, opened a new log
I20260812 11:11:09.274936 27011 ts_tablet_manager.cc:1397] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 11:11:09.275403 27011 raft_consensus.cc:348] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.275455 27009 tablet_bootstrap.cc:492] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: No bootstrap required, opened a new log
I20260812 11:11:09.275486 27011 raft_consensus.cc:374] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.275508 27011 raft_consensus.cc:729] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c0cce09e364e2294758936e8966aed, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.275537 27009 ts_tablet_manager.cc:1397] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 11:11:09.275631 27011 consensus_queue.cc:260] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.275900 27011 ts_tablet_manager.cc:1428] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:09.275858 27019 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.275981 27019 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.275991 27009 raft_consensus.cc:348] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.276113 27009 raft_consensus.cc:374] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:09.276081 27024 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.276150 27009 raft_consensus.cc:729] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3ab6e2086d46df992a99894606e735, State: Initialized, Role: FOLLOWER
I20260812 11:11:09.276199 27024 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.276286 27009 consensus_queue.cc:260] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.276623 27017 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.276788 27017 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.276623 27009 ts_tablet_manager.cc:1428] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:09.298146 27019 raft_consensus.cc:482] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:09.298291 27019 raft_consensus.cc:504] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.299427 27019 leader_election.cc:288] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781), 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195:43451)
I20260812 11:11:09.303846 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.299781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:35517 (local address 127.23.97.195:43451)
0812 11:11:09.299916 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.299920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.300165 (+ 245us) server_negotiation.cc:407] Connection header received
0812 11:11:09.300345 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.300348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.300469 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.300555 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:09.300863 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.301411 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.302586 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.303215 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.303367 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.303429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.303634 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.303712 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:11:09.303874 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.299677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43451 (local address 127.23.97.193:35517)
0812 11:11:09.300067 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.300080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.300225 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.300572 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.300575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.300594 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:09.300747 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.300753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.301535 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.301539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.302458 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.302468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.303359 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.303363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.303501 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.303531 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.303591 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.303686 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":42,"thread_start_us":154,"threads_started":1}
I20260812 11:11:09.303903 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.299778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:42625 (local address 127.23.97.194:32781)
0812 11:11:09.299912 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.299918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.300408 (+ 490us) server_negotiation.cc:407] Connection header received
0812 11:11:09.300545 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.300548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.300598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.300684 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:09.300959 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.301473 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.302663 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.303229 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.303355 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.303412 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.303700 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.303789 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:09.303875 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.299677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:32781 (local address 127.23.97.193:42625)
0812 11:11:09.300273 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.300286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.300425 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.300699 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.300702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.300720 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:09.300851 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.300856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.301613 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.301617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.302487 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.302499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.303364 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.303368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.303537 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.303555 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.303653 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.303717 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":311,"thread_start_us":118,"threads_started":1}
I20260812 11:11:09.304512 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972f1f1cf5b34b8f8caef910317bac9c" candidate_uuid: "81c0cce09e364e2294758936e8966aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735" is_pre_election: true
I20260812 11:11:09.304679 26642 raft_consensus.cc:2454] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81c0cce09e364e2294758936e8966aed in term 0.
I20260812 11:11:09.304602 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972f1f1cf5b34b8f8caef910317bac9c" candidate_uuid: "81c0cce09e364e2294758936e8966aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" is_pre_election: true
I20260812 11:11:09.304878 26877 raft_consensus.cc:2454] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81c0cce09e364e2294758936e8966aed in term 0.
I20260812 11:11:09.305047 26249 leader_election.cc:302] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3ab6e2086d46df992a99894606e735, 81c0cce09e364e2294758936e8966aed; no voters:
I20260812 11:11:09.305339 27019 raft_consensus.cc:2781] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:09.305397 27019 raft_consensus.cc:482] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:09.305462 27019 raft_consensus.cc:3037] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.306645 27019 raft_consensus.cc:504] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.307015 27019 leader_election.cc:288] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [CANDIDATE]: Term 1 election: Requested vote from peers 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781), 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195:43451)
I20260812 11:11:09.307360 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972f1f1cf5b34b8f8caef910317bac9c" candidate_uuid: "81c0cce09e364e2294758936e8966aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3ab6e2086d46df992a99894606e735"
I20260812 11:11:09.307406 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972f1f1cf5b34b8f8caef910317bac9c" candidate_uuid: "81c0cce09e364e2294758936e8966aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1"
I20260812 11:11:09.307515 26642 raft_consensus.cc:3037] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.307536 26877 raft_consensus.cc:3037] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:09.308512 26642 raft_consensus.cc:2454] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81c0cce09e364e2294758936e8966aed in term 1.
I20260812 11:11:09.308552 26877 raft_consensus.cc:2454] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81c0cce09e364e2294758936e8966aed in term 1.
I20260812 11:11:09.308780 26249 leader_election.cc:302] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3ab6e2086d46df992a99894606e735, 81c0cce09e364e2294758936e8966aed; no voters:
I20260812 11:11:09.308954 27019 raft_consensus.cc:2781] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:09.309216 27019 raft_consensus.cc:686] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [term 1 LEADER]: Becoming Leader. State: Replica: 81c0cce09e364e2294758936e8966aed, State: Running, Role: LEADER
I20260812 11:11:09.309499 27019 consensus_queue.cc:237] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } }
I20260812 11:11:09.310977 27084 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: TxnStatusTablet state changed. Reason: New leader 81c0cce09e364e2294758936e8966aed. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.311108 27084 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:09.311861 27088 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:11:09.312071 27088 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:11:09.312641 26106 catalog_manager.cc:5495] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed reported cstate change: term changed from 0 to 1, leader changed from <none> to 81c0cce09e364e2294758936e8966aed (127.23.97.193). New cstate: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:09.331431 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.327199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44703 (local address 127.0.0.1:53970)
0812 11:11:09.327472 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.327485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.327576 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.327901 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.327903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.327909 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:09.328001 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.328005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.329030 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.329032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.329689 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.329699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.330266 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.330268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.330340 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.330387 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.331153 (+ 766us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.331170 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.331252 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.331323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260812 11:11:09.331457 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.327318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.23.97.193:44703)
0812 11:11:09.327456 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.327461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.327549 (+ 88us) server_negotiation.cc:407] Connection header received
0812 11:11:09.327706 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.327710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.327780 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.327885 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:09.328097 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.328897 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.329813 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.330148 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.330251 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.330522 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.331007 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.331120 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.331271 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.331330 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:09.334735 26642 raft_consensus.cc:1261] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Refusing update from remote peer 81c0cce09e364e2294758936e8966aed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:09.335012 26877 raft_consensus.cc:1261] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Refusing update from remote peer 81c0cce09e364e2294758936e8966aed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:09.335340 27019 consensus_queue.cc:1035] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:09.335510 27084 consensus_queue.cc:1035] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:09.337073 27017 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: TxnStatusTablet state changed. Reason: New leader 81c0cce09e364e2294758936e8966aed. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.337191 27017 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.337303 27024 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: TxnStatusTablet state changed. Reason: New leader 81c0cce09e364e2294758936e8966aed. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.337414 27024 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.339107 27084 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.339221 27084 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:09.339335 27030 mvcc.cc:205] Tried to move back new op lower bound from 7317639451989803008 to 7317639451894046720. Current Snapshot: MvccSnapshot[applied={T|T < 7317639451989803008}]
I20260812 11:11:09.339473 27024 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.339570 27024 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 92d8b6ba40d64690ae2d5fd34f5252c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.346799 27084 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.346922 27084 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 81c0cce09e364e2294758936e8966aed: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:09.347707 27017 tablet_replica.cc:439] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81c0cce09e364e2294758936e8966aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 } } peers { permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 } } peers { permanent_uuid: "92d8b6ba40d64690ae2d5fd34f5252c1" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43451 } } }
I20260812 11:11:09.347829 27017 tablet_replica.cc:442] T 972f1f1cf5b34b8f8caef910317bac9c P 6f3ab6e2086d46df992a99894606e735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:09.356393 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.352699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:44805 (local address 127.23.97.193:44703)
0812 11:11:09.352829 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.352833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.353072 (+ 239us) server_negotiation.cc:407] Connection header received
0812 11:11:09.353152 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.353155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.353203 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.353279 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:09.353465 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.354262 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.355118 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.355450 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.355555 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.355806 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.355886 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.355961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.356187 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.356243 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:11:09.356454 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.352616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44703 (local address 127.23.97.195:44805)
0812 11:11:09.352979 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.352991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.353081 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.353279 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.353281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.353287 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:09.353370 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.353374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.354368 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.354370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.354972 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.354980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.355560 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.355563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.355629 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.355671 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.356011 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.356019 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.356164 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.356265 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260812 11:11:09.363495 26642 raft_consensus.cc:1261] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Refusing update from remote peer 92d8b6ba40d64690ae2d5fd34f5252c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:09.363562 26375 raft_consensus.cc:1261] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Refusing update from remote peer 92d8b6ba40d64690ae2d5fd34f5252c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:09.363955 27022 consensus_queue.cc:1035] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c0cce09e364e2294758936e8966aed" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:09.364101 27024 consensus_queue.cc:1035] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3ab6e2086d46df992a99894606e735" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:09.387590 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.383106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43451 (local address 127.23.97.193:57461)
0812 11:11:09.383455 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:09.383471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:09.383567 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:09.383801 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:09.383804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:09.383812 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:09.383943 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.383949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.385176 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.385180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.386179 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:09.386192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.386706 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.386710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:09.386827 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.386875 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.387270 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.387280 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:11:09.387349 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 11:11:09.387434 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
I20260812 11:11:09.387569 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.383382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:57461 (local address 127.23.97.195:43451)
0812 11:11:09.383502 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.383507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.383526 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:09.383661 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.383663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.383700 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.383763 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:09.384045 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.385048 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.386317 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.386573 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.386697 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.387026 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:09.387133 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:09.387253 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.387362 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.387416 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:11:09.408581 26107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:09.408763 26107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:09.410887 26107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P babd588ff2a441978588dde63ebd2c14: Sending DeleteTablet for 3 replicas of tablet 572eeab8fd9f4e4d8e833f141f85579e
I20260812 11:11:09.411567 26608 tablet_service.cc:1514] Processing DeleteTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:09 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:55760
I20260812 11:11:09.411578 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:09 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260812 11:11:09.411574 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 572eeab8fd9f4e4d8e833f141f85579e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:09 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:51252
I20260812 11:11:09.411761 27041 tablet_replica.cc:330] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: stopping tablet replica
I20260812 11:11:09.411779 27046 tablet_replica.cc:330] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: stopping tablet replica
I20260812 11:11:09.411857 27041 raft_consensus.cc:2229] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:09.411888 27046 raft_consensus.cc:2229] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:09.411964 27041 raft_consensus.cc:2258] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:09.412035 27046 raft_consensus.cc:2258] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:09.412252 27050 tablet_replica.cc:330] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: stopping tablet replica
I20260812 11:11:09.412420 27050 raft_consensus.cc:2229] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:09.412552 27050 raft_consensus.cc:2258] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:09.412961 27041 ts_tablet_manager.cc:1905] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:09.413334 27046 ts_tablet_manager.cc:1905] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:09.413458 27050 ts_tablet_manager.cc:1905] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:09.416134 27046 ts_tablet_manager.cc:1918] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:09.416203 27046 log.cc:1196] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/572eeab8fd9f4e4d8e833f141f85579e
I20260812 11:11:09.416313 27050 ts_tablet_manager.cc:1918] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:09.416419 27050 log.cc:1196] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/572eeab8fd9f4e4d8e833f141f85579e
I20260812 11:11:09.416519 27046 ts_tablet_manager.cc:1939] T 572eeab8fd9f4e4d8e833f141f85579e P 92d8b6ba40d64690ae2d5fd34f5252c1: Deleting consensus metadata
I20260812 11:11:09.416736 27050 ts_tablet_manager.cc:1939] T 572eeab8fd9f4e4d8e833f141f85579e P 81c0cce09e364e2294758936e8966aed: Deleting consensus metadata
I20260812 11:11:09.416747 27041 ts_tablet_manager.cc:1918] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:09.416822 27041 log.cc:1196] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/572eeab8fd9f4e4d8e833f141f85579e
I20260812 11:11:09.417110 27041 ts_tablet_manager.cc:1939] T 572eeab8fd9f4e4d8e833f141f85579e P 6f3ab6e2086d46df992a99894606e735: Deleting consensus metadata
I20260812 11:11:09.417192 26074 catalog_manager.cc:4841] TS 92d8b6ba40d64690ae2d5fd34f5252c1 (127.23.97.195:43451): tablet 572eeab8fd9f4e4d8e833f141f85579e (table test-table [id=dd1648f3548e47178e22514902ddf5f8]) successfully deleted
I20260812 11:11:09.417528 26072 catalog_manager.cc:4841] TS 81c0cce09e364e2294758936e8966aed (127.23.97.193:44703): tablet 572eeab8fd9f4e4d8e833f141f85579e (table test-table [id=dd1648f3548e47178e22514902ddf5f8]) successfully deleted
I20260812 11:11:09.417928 26072 catalog_manager.cc:4841] TS 6f3ab6e2086d46df992a99894606e735 (127.23.97.194:32781): tablet 572eeab8fd9f4e4d8e833f141f85579e (table test-table [id=dd1648f3548e47178e22514902ddf5f8]) successfully deleted
Aug 12 11:11:09 dist-test-slave-zpvx krb5kdc[26020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533068, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:09.431432 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.419268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.23.97.193:44703)
0812 11:11:09.419412 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.419420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.419743 (+ 323us) server_negotiation.cc:407] Connection header received
0812 11:11:09.419936 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.419939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.420020 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.420104 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:09.421022 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.421575 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.422764 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.422974 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.423065 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.427028 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:09.427039 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:09.427051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:09.427086 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:09.429285 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.429841 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.429848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.429850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.429958 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.430293 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.430297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.430299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.430636 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:09.430855 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.431111 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.431260 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 11:11:09.432075 26453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53980
I20260812 11:11:09.450621 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:09.440826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.23.97.254:38425)
0812 11:11:09.440967 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:09.440974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:09.441138 (+ 164us) server_negotiation.cc:407] Connection header received
0812 11:11:09.441281 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:09.441284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:09.441339 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:09.441434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:09.442304 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.443148 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.444461 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:09.444833 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:09.444952 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:09.446708 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:09.446714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:09.446719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:09.446752 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:09.448625 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.449132 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.449136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.449138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.449245 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:09.449568 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:09.449571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:09.449573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:09.449829 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:09.449982 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:09.450311 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 11:11:09.450456 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 11:11:09.451047 26107 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35848
I20260812 11:11:09.452737 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26222
I20260812 11:11:09.474256 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26479
I20260812 11:11:09.492820 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26735
I20260812 11:11:09.505731 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26046
2026-08-12T11:11:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3179 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:09 dist-test-slave-zpvx krb5kdc[27130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:09 dist-test-slave-zpvx krb5kdc[27130](info): set up 2 sockets
Aug 12 11:11:09 dist-test-slave-zpvx krb5kdc[27130](info): commencing operation
krb5kdc: starting...
W20260812 11:11:10.820434 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:10 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:10.941738 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:45609
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:45609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:11.051945 27149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:11.052238 27149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:11.052320 27149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:11.056262 27149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:11.056345 27149 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:11:11.056380 27149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:11.056438 27149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:11.056469 27149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:11.056501 27149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:11.061587 27149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:45609
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:11.062570 27149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:11.063563 27149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:11.072219 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.072162 27159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.072145 27164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:11.072271 27149 server_base.cc:1034] running on GCE node
I20260812 11:11:11.072934 27149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:11.073998 27149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:11.075186 27149 hybrid_clock.cc:648] HybridClock initialized: now 1786533071075166 us; error 42 us; skew 500 ppm
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:11.078006 27149 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:11.078889 27149 webserver.cc:460] Webserver started at http://127.23.97.254:33927/ using document root <none> and password file <none>
I20260812 11:11:11.079425 27149 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:11.079514 27149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:11.079754 27149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:11.081784 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "e049c9d8ea3744d9aa9457047c11f185"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "03758a5dabd29099d70515b8a77ac5a6"
server_key_iv: "23654631681bc4cc572ed5fc9cb4d71e"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.082347 27149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "e049c9d8ea3744d9aa9457047c11f185"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "03758a5dabd29099d70515b8a77ac5a6"
server_key_iv: "23654631681bc4cc572ed5fc9cb4d71e"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.085979 27149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260812 11:11:11.088243 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.089529 27149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:11.089697 27149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e049c9d8ea3744d9aa9457047c11f185"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "03758a5dabd29099d70515b8a77ac5a6"
server_key_iv: "23654631681bc4cc572ed5fc9cb4d71e"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.089813 27149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:11.113260 27149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:11.116688 27149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:11.116899 27149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:11.124938 27149 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:45609
I20260812 11:11:11.124956 27274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:45609 every 8 connection(s)
I20260812 11:11:11.125874 27149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:11.127774 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27149
I20260812 11:11:11.127888 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:11.128158 23943 external_mini_cluster.cc:1428] Setting key 295fa07781f8bab3fd2f3f928d50ef8c
I20260812 11:11:11.128968 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:11.135347 27277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Bootstrap starting.
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:11.137800 27277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:11.138957 27277 log.cc:824] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:11.140900 27277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: No bootstrap required, opened a new log
I20260812 11:11:11.142769 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.129535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.23.97.254:45609)
0812 11:11:11.129948 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.129974 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.130009 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:11.130625 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.130649 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.130950 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.131219 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.132110 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.133168 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.134357 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.134828 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.134945 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.137753 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.137758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.137771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.137798 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.139885 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.140384 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.140393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.140398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.140541 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.140837 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.140840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.140842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.141285 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.141489 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.141797 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.142081 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260812 11:11:11.144073 27277 raft_consensus.cc:348] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } }
I20260812 11:11:11.144336 27277 raft_consensus.cc:374] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:11.144429 27277 raft_consensus.cc:729] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e049c9d8ea3744d9aa9457047c11f185, State: Initialized, Role: FOLLOWER
I20260812 11:11:11.144940 27277 consensus_queue.cc:260] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } }
I20260812 11:11:11.145107 27277 raft_consensus.cc:388] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:11.145175 27277 raft_consensus.cc:482] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:11.145251 27277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:11.146041 27277 raft_consensus.cc:504] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } }
I20260812 11:11:11.146371 27277 leader_election.cc:302] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e049c9d8ea3744d9aa9457047c11f185; no voters:
I20260812 11:11:11.146618 27277 leader_election.cc:288] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:11.146770 27295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:11.147017 27295 raft_consensus.cc:686] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [term 1 LEADER]: Becoming Leader. State: Replica: e049c9d8ea3744d9aa9457047c11f185, State: Running, Role: LEADER
I20260812 11:11:11.147317 27295 consensus_queue.cc:237] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } }
I20260812 11:11:11.147894 27277 sys_catalog.cc:564] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:11.149188 27299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e049c9d8ea3744d9aa9457047c11f185. Latest consensus state: current_term: 1 leader_uuid: "e049c9d8ea3744d9aa9457047c11f185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } } }
I20260812 11:11:11.149339 27299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:11.149611 27296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e049c9d8ea3744d9aa9457047c11f185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e049c9d8ea3744d9aa9457047c11f185" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45609 } } }
I20260812 11:11:11.149725 27296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:11.149804 27308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:11.152869 27308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:11.158093 27308 catalog_manager.cc:1347] Generated new cluster ID: cd70cfeaa1274cccbafc1097cb03c1a0
I20260812 11:11:11.158170 27308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:11.180982 27308 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:11.181907 27308 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:11.192341 27308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Generated new TSK 0
I20260812 11:11:11.193013 27308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:11.230839 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45609
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:11.348794 27333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:11.349088 27333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:11.349140 27333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:11.353268 27333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:11.353340 27333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:11.353381 27333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:11:11.353396 27333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:11.353432 27333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:11.358667 27333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:45609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:11.359717 27333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:11.360733 27333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:11.370905 27345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:11.370968 27333 server_base.cc:1034] running on GCE node
W20260812 11:11:11.370913 27348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.371031 27344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:11.371490 27333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:11.372151 27333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:11.373353 27333 hybrid_clock.cc:648] HybridClock initialized: now 1786533071373341 us; error 55 us; skew 500 ppm
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:11.376178 27333 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:11.377156 27333 webserver.cc:460] Webserver started at http://127.23.97.193:43909/ using document root <none> and password file <none>
I20260812 11:11:11.377718 27333 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:11.377784 27333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:11.377960 27333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:11.380007 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "dce60d7f17f3a489d5a67d0d050055d7"
server_key_iv: "c2dcbd29371a464e48bf3e6092680c42"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.380614 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "dce60d7f17f3a489d5a67d0d050055d7"
server_key_iv: "c2dcbd29371a464e48bf3e6092680c42"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.384449 27333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 11:11:11.386940 27361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.388267 27333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:11:11.388427 27333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "dce60d7f17f3a489d5a67d0d050055d7"
server_key_iv: "c2dcbd29371a464e48bf3e6092680c42"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.388548 27333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:11.404788 27333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:11.408097 27333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:11.408325 27333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:11.409924 27333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:11.409976 27333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.410044 27333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:11.410079 27333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:11.421766 27333 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:36155
I20260812 11:11:11.421837 27553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:36155 every 8 connection(s)
I20260812 11:11:11.422993 27333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:11.425619 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.410893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:45733 (local address 127.23.97.254:45609)
0812 11:11:11.411025 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.411028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.412168 (+ 1140us) server_negotiation.cc:407] Connection header received
0812 11:11:11.413384 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.413388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.413449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.413549 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.415313 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.416163 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.418206 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.418443 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.418528 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.421845 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.421850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.421853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.421884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.423550 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.424133 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.424136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.424137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.424246 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.424623 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.424626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.424627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.424820 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.424898 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.425381 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.425497 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:11.426396 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.411491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.193:45733)
0812 11:11:11.412010 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.412054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.413065 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.413788 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.413803 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.414348 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.415103 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.415127 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.416320 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.416327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.418037 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.418055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.418632 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.418638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.418865 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.419598 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.419625 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.421643 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.423697 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.423705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.423723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.424038 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.424324 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.424326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.424327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.424529 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.424911 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.424916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.425142 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.425314 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.425570 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":155,"threads_started":1}
I20260812 11:11:11.427803 27554 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45609
I20260812 11:11:11.427951 27554 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:11.428104 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27333
I20260812 11:11:11.428228 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:11.428555 23943 external_mini_cluster.cc:1428] Setting key f6cc27553dd98ea3ff8c57272f2a7ffd
I20260812 11:11:11.428551 27554 heartbeater.cc:511] Master 127.23.97.254:45609 requested a full tablet report, sending...
I20260812 11:11:11.430199 27203 ts_manager.cc:194] Registered new tserver with Master: 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155)
I20260812 11:11:11.431598 27203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:45733
I20260812 11:11:11.439035 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.430172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.193:38197)
0812 11:11:11.430629 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.430644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.430760 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.431010 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.431012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.431264 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.431430 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.431442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.432416 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.432420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.433347 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.433359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.434059 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.434064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.434173 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.434796 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.434818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.435212 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.437196 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.437202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.437204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.437437 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.437768 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.437772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.437774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.437872 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.438307 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.438311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.438427 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.438619 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.438798 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260812 11:11:11.439025 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.430242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:38197 (local address 127.23.97.254:45609)
0812 11:11:11.430473 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.430478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.430736 (+ 258us) server_negotiation.cc:407] Connection header received
0812 11:11:11.430840 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.430843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.430886 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.430973 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.431566 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.432260 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.433483 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.433883 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.434009 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.435377 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.435381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.435385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.435412 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.437036 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.437563 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.437567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.437569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.437662 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.437976 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.437981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.437983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.438196 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.438318 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.438711 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.438843 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":147}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:11.459827 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45609
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:11.576004 27576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:11.576273 27576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:11.576355 27576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:11.580389 27576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:11.580470 27576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:11.580546 27576 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:11:11.580597 27576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:11.580657 27576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:11.585666 27576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:45609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:11.586615 27576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:11.587554 27576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:11.597174 27585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.597173 27587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.597234 27584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:11.597689 27576 server_base.cc:1034] running on GCE node
I20260812 11:11:11.598102 27576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:11.598716 27576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:11.599880 27576 hybrid_clock.cc:648] HybridClock initialized: now 1786533071599848 us; error 43 us; skew 500 ppm
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:11.602717 27576 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:11.603646 27576 webserver.cc:460] Webserver started at http://127.23.97.194:45073/ using document root <none> and password file <none>
I20260812 11:11:11.604218 27576 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:11.604315 27576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:11.604575 27576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:11.606614 27576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "771d28a868384ebe8d1d375057badaa2"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "72d4286fed15ec030a41308b26cdad98"
server_key_iv: "4aafd668ed020e6162480e0ed7244786"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.607180 27576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "771d28a868384ebe8d1d375057badaa2"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "72d4286fed15ec030a41308b26cdad98"
server_key_iv: "4aafd668ed020e6162480e0ed7244786"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.610989 27576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:11:11.613493 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.614693 27576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:11.614830 27576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "771d28a868384ebe8d1d375057badaa2"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "72d4286fed15ec030a41308b26cdad98"
server_key_iv: "4aafd668ed020e6162480e0ed7244786"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.614933 27576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:11.639310 27576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:11.642743 27576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:11.642973 27576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:11.644512 27576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:11.644564 27576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.644632 27576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:11.644665 27576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:11.656179 27576 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:45551
I20260812 11:11:11.656364 27797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:45551 every 8 connection(s)
I20260812 11:11:11.657186 27576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:11.659458 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.645336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:44331 (local address 127.23.97.254:45609)
0812 11:11:11.645512 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.645519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.646339 (+ 820us) server_negotiation.cc:407] Connection header received
0812 11:11:11.647433 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.647438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.647498 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.647560 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.649463 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.650151 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.652065 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.652469 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.652595 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.655659 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.655667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.655672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.655704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.657459 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.658053 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.658057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.658058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.658141 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.658468 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.658470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.658471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.658648 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.658732 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.659231 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.659337 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:11:11.660277 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.645709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.194:44331)
0812 11:11:11.646158 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.646208 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.647172 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.648024 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.648035 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.648577 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.649313 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.649335 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.650323 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.650329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.651920 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.651936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.652669 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.652677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.652906 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.653522 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.653546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.655473 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.657579 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.657585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.657598 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.657945 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.658239 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.658249 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.658251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.658379 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.658772 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.658778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.659001 (+ 223us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.659181 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.659434 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260812 11:11:11.661444 27798 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45609
I20260812 11:11:11.661621 27798 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:11.662112 27798 heartbeater.cc:511] Master 127.23.97.254:45609 requested a full tablet report, sending...
I20260812 11:11:11.663204 27203 ts_manager.cc:194] Registered new tserver with Master: 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:11.663937 27203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:44331
I20260812 11:11:11.667286 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27576
I20260812 11:11:11.667411 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:11.667769 23943 external_mini_cluster.cc:1428] Setting key 58fe0245c73fc629206b1aa10ce787b2
I20260812 11:11:11.673249 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.663632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:40855 (local address 127.23.97.254:45609)
0812 11:11:11.663740 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.663744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.664457 (+ 713us) server_negotiation.cc:407] Connection header received
0812 11:11:11.664596 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.664599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.664646 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.664735 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.665165 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.665890 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.666690 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.666897 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.666985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.668142 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.668148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.668154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.668182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.671558 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.671996 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.672000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.672001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.672084 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.672383 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.672390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.672394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.672602 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.672714 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.672983 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.673094 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:11:11.673257 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.663557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.194:40855)
0812 11:11:11.664351 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.664393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.664492 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.664756 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.664758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.664927 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.665023 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.665028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.666025 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.666028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.666565 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.666573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.667002 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.667005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.667090 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.667572 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.667590 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.667992 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.671690 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.671694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.671696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.671893 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.672184 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.672187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.672189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.672273 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.672709 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.672712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.672808 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.672914 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.673049 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:11.700129 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45609
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:11.821941 27815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:11.822219 27815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:11.822297 27815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:11.826212 27815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:11.826287 27815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:11.826364 27815 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:11:11.826404 27815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:11.826470 27815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:11.831619 27815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:45609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:11.832602 27815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:11.833673 27815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:11.844667 27822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.844676 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:11.844645 27823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:11.844944 27815 server_base.cc:1034] running on GCE node
I20260812 11:11:11.845355 27815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:11.845954 27815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:11.847143 27815 hybrid_clock.cc:648] HybridClock initialized: now 1786533071847124 us; error 31 us; skew 500 ppm
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:11.850258 27815 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:11.851076 27815 webserver.cc:460] Webserver started at http://127.23.97.195:35505/ using document root <none> and password file <none>
I20260812 11:11:11.851645 27815 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:11.851742 27815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:11.851979 27815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:11.854029 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "35a043e07332486299783b33fdfbb070"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "a866d3e8e2e192fd2869873e36622f52"
server_key_iv: "2e50cd58510847ec81fd31c88ada4e2d"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.854615 27815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "35a043e07332486299783b33fdfbb070"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "a866d3e8e2e192fd2869873e36622f52"
server_key_iv: "2e50cd58510847ec81fd31c88ada4e2d"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.858402 27815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:11:11.860980 27841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.862262 27815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:11.862399 27815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "35a043e07332486299783b33fdfbb070"
format_stamp: "Formatted at 2026-08-12 11:11:11 on dist-test-slave-zpvx"
server_key: "a866d3e8e2e192fd2869873e36622f52"
server_key_iv: "2e50cd58510847ec81fd31c88ada4e2d"
server_key_version: "encryptionkey@0"
I20260812 11:11:11.862514 27815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:11.873984 27815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:11.876845 27815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:11.877044 27815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:11.878443 27815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:11.878520 27815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:11.878592 27815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:11.878644 27815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:11.889798 27815 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:33107
I20260812 11:11:11.890239 28030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:33107 every 8 connection(s)
I20260812 11:11:11.890893 27815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:11.893657 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.879294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:43873 (local address 127.23.97.254:45609)
0812 11:11:11.879463 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.879470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.880763 (+ 1293us) server_negotiation.cc:407] Connection header received
0812 11:11:11.881975 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.881979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.882047 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.882160 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.883437 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.884077 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.885958 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.886335 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.886450 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.889346 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.889352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.889357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.889391 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.891136 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.891759 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.891763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.891765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.891870 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.892261 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.892264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.892265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.892582 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.892702 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.893369 (+ 667us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.893526 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:11:11.894451 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.880117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.195:43873)
0812 11:11:11.880588 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.880637 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.881712 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.882305 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.882312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.882813 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.883286 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.883299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.884208 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.884213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.885818 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.885834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.886601 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.886604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.886776 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.887355 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.887371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.889168 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.891258 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.891278 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.891298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.891645 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.891970 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.891972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.891974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.892150 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.892764 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.892769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.893118 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.893278 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.893525 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260812 11:11:11.895776 28031 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45609
I20260812 11:11:11.895987 28031 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:11.896597 28031 heartbeater.cc:511] Master 127.23.97.254:45609 requested a full tablet report, sending...
I20260812 11:11:11.897694 27202 ts_manager.cc:194] Registered new tserver with Master: 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107)
I20260812 11:11:11.898157 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27815
I20260812 11:11:11.898285 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:11.898602 23943 external_mini_cluster.cc:1428] Setting key 824cf9c2c8cbb8d70243ad141c480578
I20260812 11:11:11.898599 27202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:43873
I20260812 11:11:11.900839 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:11:11.905418 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.897933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:48233 (local address 127.23.97.254:45609)
0812 11:11:11.898043 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.898047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.898274 (+ 227us) server_negotiation.cc:407] Connection header received
0812 11:11:11.898425 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.898428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.898468 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.898563 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.899034 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.899765 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.900899 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.901133 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.901243 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.902323 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.902327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.902330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.902354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.903768 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.904189 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.904193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.904195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.904279 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.904601 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.904605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.904607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.904788 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.904897 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.905152 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.905262 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:11:11.905436 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.897879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.23.97.195:48233)
0812 11:11:11.898177 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.898189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.898343 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.898579 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.898582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.898775 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:11.898900 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.898912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.899930 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.899933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.900770 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.900783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.901254 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.901258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.901353 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.901870 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:11.901882 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:11.902188 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:11.903884 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.903889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.903891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.904089 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.904410 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:11.904414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:11.904419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:11.904508 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:11.904888 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:11.904891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:11.904989 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.905095 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.905250 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:11:11 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:11.928921 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.921032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.23.97.254:45609)
0812 11:11:11.921164 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.921170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.921189 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:11.921226 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.921228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.921266 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.921336 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:11.922014 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.922537 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.923327 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.923534 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.923622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.926042 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:11.926045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:11.926047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:11.926070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:11.927317 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.927689 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.927692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.927693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.927763 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:11.928048 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:11.928055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:11.928056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:11.928220 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:11.928320 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.928606 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.928699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:11:11.931814 27202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:11.934983 27202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:11.949327 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.943278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.0.0.1:41362)
0812 11:11:11.943605 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.943651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.943768 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.944970 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.944976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.945002 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:11.945128 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.945133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.946404 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.946409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.947630 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.947647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.948777 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.948781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.948892 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.948934 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.949048 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.949183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260812 11:11:11.949354 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.943448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.23.97.195:33107)
0812 11:11:11.944202 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.944217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.944245 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:11:11.944328 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.944333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.944591 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.944779 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:11.945261 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.946294 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.947755 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.948613 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.948785 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.948952 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.949090 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.949185 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":111,"threads_started":1}
I20260812 11:11:11.950776 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.944025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36155 (local address 127.0.0.1:34326)
0812 11:11:11.944827 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.944838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.944950 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.945426 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.945429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.945447 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:11.945541 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.945547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.946730 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.946734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.947894 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.947905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.949279 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.949283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.950445 (+ 1162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.950474 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.950571 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.950643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":86,"threads_started":1}
I20260812 11:11:11.950829 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.944209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.23.97.193:36155)
0812 11:11:11.944684 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.944692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.944932 (+ 240us) server_negotiation.cc:407] Connection header received
0812 11:11:11.945064 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.945071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.945294 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.945457 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:11.945649 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.946594 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.948023 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.948912 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.949079 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.949213 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.950605 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.950698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":74,"threads_started":1}
I20260812 11:11:11.951519 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.944110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45551 (local address 127.0.0.1:39196)
0812 11:11:11.945133 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:11.945146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:11.945224 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:11.945849 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:11.945852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:11.945869 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:11.945958 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.945964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.947502 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.947505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.948502 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:11.948511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.949898 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.949901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:11.951202 (+ 1301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.951221 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:11:11.951316 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 11:11:11.951384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":1222,"thread_start_us":101,"threads_started":1}
I20260812 11:11:11.951576 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:11.944579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.23.97.194:45551)
0812 11:11:11.944993 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:11.945004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:11.945292 (+ 288us) server_negotiation.cc:407] Connection header received
0812 11:11:11.945380 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:11.945385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:11.945571 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:11.945714 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:11.946093 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.947379 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.948716 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:11.949545 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:11.949650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:11.949827 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:11.951351 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0812 11:11:11.951438 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":113,"threads_started":1}
I20260812 11:11:11.952801 27923 tablet_service.cc:1467] Processing CreateTablet for tablet 622195d4e2b24993acd7676efefe9f93 (DEFAULT_TABLE table=test-table [id=738c37eb1b3443de93fc82ace1077ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:11.953850 27923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622195d4e2b24993acd7676efefe9f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:11.953756 27440 tablet_service.cc:1467] Processing CreateTablet for tablet 622195d4e2b24993acd7676efefe9f93 (DEFAULT_TABLE table=test-table [id=738c37eb1b3443de93fc82ace1077ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:11.954315 27680 tablet_service.cc:1467] Processing CreateTablet for tablet 622195d4e2b24993acd7676efefe9f93 (DEFAULT_TABLE table=test-table [id=738c37eb1b3443de93fc82ace1077ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:11.954792 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622195d4e2b24993acd7676efefe9f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:11.955255 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622195d4e2b24993acd7676efefe9f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:11.960530 28067 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Bootstrap starting.
I20260812 11:11:11.961866 28069 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Bootstrap starting.
I20260812 11:11:11.962890 28067 tablet_bootstrap.cc:654] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:11.963124 28071 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Bootstrap starting.
I20260812 11:11:11.963815 28067 log.cc:824] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:11.964002 28069 tablet_bootstrap.cc:654] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:11.964879 28069 log.cc:824] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:11.965072 28071 tablet_bootstrap.cc:654] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:11.965831 28071 log.cc:824] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:11.966146 28067 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: No bootstrap required, opened a new log
I20260812 11:11:11.966351 28067 ts_tablet_manager.cc:1397] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:11.966797 28069 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: No bootstrap required, opened a new log
I20260812 11:11:11.966981 28069 ts_tablet_manager.cc:1397] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:11:11.967824 28071 tablet_bootstrap.cc:492] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: No bootstrap required, opened a new log
I20260812 11:11:11.968012 28071 ts_tablet_manager.cc:1397] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:11:11.969705 28067 raft_consensus.cc:348] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.969930 28067 raft_consensus.cc:374] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:11.969985 28067 raft_consensus.cc:729] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a043e07332486299783b33fdfbb070, State: Initialized, Role: FOLLOWER
I20260812 11:11:11.970412 28067 consensus_queue.cc:260] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.971053 28031 heartbeater.cc:503] Master 127.23.97.254:45609 was elected leader, sending a full tablet report...
I20260812 11:11:11.971457 28067 ts_tablet_manager.cc:1428] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:11.971486 28069 raft_consensus.cc:348] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.971717 28069 raft_consensus.cc:374] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:11.971776 28069 raft_consensus.cc:729] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Initialized, Role: FOLLOWER
I20260812 11:11:11.972205 28069 consensus_queue.cc:260] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.972831 27554 heartbeater.cc:503] Master 127.23.97.254:45609 was elected leader, sending a full tablet report...
I20260812 11:11:11.973126 28069 ts_tablet_manager.cc:1428] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:11:11.973026 28071 raft_consensus.cc:348] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.973258 28071 raft_consensus.cc:374] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:11.973310 28071 raft_consensus.cc:729] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Initialized, Role: FOLLOWER
I20260812 11:11:11.973702 28071 consensus_queue.cc:260] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:11.974659 28071 ts_tablet_manager.cc:1428] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:11:11.974671 27798 heartbeater.cc:503] Master 127.23.97.254:45609 was elected leader, sending a full tablet report...
I20260812 11:11:12.132498 28083 raft_consensus.cc:482] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:12.132737 28083 raft_consensus.cc:504] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:12.133875 28083 leader_election.cc:288] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107), 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155)
I20260812 11:11:12.139118 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.134256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:37729 (local address 127.23.97.195:33107)
0812 11:11:12.134401 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.134407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.134655 (+ 248us) server_negotiation.cc:407] Connection header received
0812 11:11:12.134737 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.134740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.134799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.134881 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:12.135186 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.135994 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.137798 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.138442 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.138570 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.138626 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.138946 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.139010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:11:12.139205 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.134143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.23.97.194:37729)
0812 11:11:12.134432 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.134450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.134556 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.134895 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.134898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.134915 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:12.135041 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.135048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.136136 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.136139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.137648 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.137657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.138569 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.138572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.138770 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.138802 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.138918 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.139024 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":147}
I20260812 11:11:12.139653 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.134638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:54103 (local address 127.23.97.193:36155)
0812 11:11:12.134769 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.134773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.134826 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:11:12.135033 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.135037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.135098 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.135174 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:12.135466 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.135970 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.137798 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.138436 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.138570 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.138631 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.139423 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.139517 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:12.139925 27954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622195d4e2b24993acd7676efefe9f93" candidate_uuid: "771d28a868384ebe8d1d375057badaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a043e07332486299783b33fdfbb070" is_pre_election: true
I20260812 11:11:12.140089 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.134435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36155 (local address 127.23.97.194:54103)
0812 11:11:12.134736 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.134749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.134899 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.135195 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.135198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.135214 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:12.135345 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.135350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.136095 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.136101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.137645 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.137657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.138569 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.138572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.139255 (+ 683us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.139284 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.139862 (+ 578us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.139946 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":154,"thread_start_us":79,"threads_started":1}
I20260812 11:11:12.140271 27954 raft_consensus.cc:2454] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771d28a868384ebe8d1d375057badaa2 in term 0.
I20260812 11:11:12.140841 27605 leader_election.cc:302] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:12.141112 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622195d4e2b24993acd7676efefe9f93" candidate_uuid: "771d28a868384ebe8d1d375057badaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" is_pre_election: true
I20260812 11:11:12.141171 28083 raft_consensus.cc:2781] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:12.141256 28083 raft_consensus.cc:482] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:12.141304 28083 raft_consensus.cc:3037] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.141418 27476 raft_consensus.cc:2454] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771d28a868384ebe8d1d375057badaa2 in term 0.
W20260812 11:11:12.142032 28034 tablet.cc:2369] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:12.142527 28083 raft_consensus.cc:504] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:12.142866 28083 leader_election.cc:288] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107), 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155)
I20260812 11:11:12.143180 27954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622195d4e2b24993acd7676efefe9f93" candidate_uuid: "771d28a868384ebe8d1d375057badaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a043e07332486299783b33fdfbb070"
I20260812 11:11:12.143309 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622195d4e2b24993acd7676efefe9f93" candidate_uuid: "771d28a868384ebe8d1d375057badaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
I20260812 11:11:12.143345 27954 raft_consensus.cc:3037] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.143426 27476 raft_consensus.cc:3037] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.144488 27476 raft_consensus.cc:2454] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771d28a868384ebe8d1d375057badaa2 in term 1.
I20260812 11:11:12.144830 27605 leader_election.cc:302] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3ac28c47bb4feeb8059f932c10ce7a, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:12.144935 27954 raft_consensus.cc:2454] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771d28a868384ebe8d1d375057badaa2 in term 1.
I20260812 11:11:12.145032 28083 raft_consensus.cc:2781] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:12.145325 28083 raft_consensus.cc:686] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Running, Role: LEADER
I20260812 11:11:12.145659 28083 consensus_queue.cc:237] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } }
I20260812 11:11:12.148856 27202 catalog_manager.cc:5495] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771d28a868384ebe8d1d375057badaa2 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "771d28a868384ebe8d1d375057badaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } health_report { overall_health: HEALTHY } } }
W20260812 11:11:12.158592 27799 tablet.cc:2369] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:11:12.174472 27559 tablet.cc:2369] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:11:12 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:12.214375 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.206133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45609 (local address 127.0.0.1:46608)
0812 11:11:12.206429 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.206446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.206563 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.206816 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.206818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.207228 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:12.207323 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.207329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.208082 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.208084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.208672 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.208679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.209110 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.209113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.209173 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.209781 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:12.209793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:12.211208 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:12.212921 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:12.212930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:12.212935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:12.213162 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:12.213418 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:12.213420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:12.213422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:12.213500 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:12.213889 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:12.213894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:12.213989 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.214081 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.214186 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":125,"threads_started":1}
I20260812 11:11:12.214373 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.206241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.23.97.254:45609)
0812 11:11:12.206374 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.206378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.206533 (+ 155us) server_negotiation.cc:407] Connection header received
0812 11:11:12.206646 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.206649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.206700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.206777 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:12.207461 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.207985 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.208782 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.208992 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.209109 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.211363 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:12.211366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:12.211368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:12.211389 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:12.212821 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:12.213248 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:12.213251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:12.213252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:12.213338 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:12.213607 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:12.213610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:12.213611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:12.213793 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:12.213876 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.214147 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.214244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:11:12.218370 27202 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:46608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:11:12.219151 27202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:12.223980 27680 tablet_service.cc:1467] Processing CreateTablet for tablet cdbc8051892c4b12ba290c03bb629a3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92e9d83a09b8439b9e55e04191fd75a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:12.223948 27440 tablet_service.cc:1467] Processing CreateTablet for tablet cdbc8051892c4b12ba290c03bb629a3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92e9d83a09b8439b9e55e04191fd75a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:12.224663 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbc8051892c4b12ba290c03bb629a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.224685 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbc8051892c4b12ba290c03bb629a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.224330 27923 tablet_service.cc:1467] Processing CreateTablet for tablet cdbc8051892c4b12ba290c03bb629a3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92e9d83a09b8439b9e55e04191fd75a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:12.224946 27923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbc8051892c4b12ba290c03bb629a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.227077 28071 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: Bootstrap starting.
I20260812 11:11:12.227774 28069 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: Bootstrap starting.
I20260812 11:11:12.227785 28067 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: Bootstrap starting.
I20260812 11:11:12.228080 28071 tablet_bootstrap.cc:654] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.228838 28069 tablet_bootstrap.cc:654] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.228917 28067 tablet_bootstrap.cc:654] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.229486 28071 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: No bootstrap required, opened a new log
I20260812 11:11:12.229578 28071 ts_tablet_manager.cc:1397] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:12.230055 28071 raft_consensus.cc:348] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.230140 28071 raft_consensus.cc:374] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.230161 28071 raft_consensus.cc:729] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.230188 28069 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: No bootstrap required, opened a new log
I20260812 11:11:12.230269 28069 ts_tablet_manager.cc:1397] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:12.230305 28067 tablet_bootstrap.cc:492] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: No bootstrap required, opened a new log
I20260812 11:11:12.230284 28071 consensus_queue.cc:260] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.230379 28067 ts_tablet_manager.cc:1397] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:12.230603 28071 ts_tablet_manager.cc:1428] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:12.230557 28083 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.230679 28083 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.230749 28069 raft_consensus.cc:348] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.230777 28067 raft_consensus.cc:348] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.230876 28067 raft_consensus.cc:374] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.230876 28069 raft_consensus.cc:374] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.230921 28067 raft_consensus.cc:729] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a043e07332486299783b33fdfbb070, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.230921 28069 raft_consensus.cc:729] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.231048 28069 consensus_queue.cc:260] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.231048 28067 consensus_queue.cc:260] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.231340 28067 ts_tablet_manager.cc:1428] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:12.231298 28079 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.231429 28079 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.231673 28082 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.231812 28082 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.232439 28069 ts_tablet_manager.cc:1428] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:12.410851 28079 raft_consensus.cc:482] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:12.410993 28079 raft_consensus.cc:504] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.412050 28079 leader_election.cc:288] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:12.416180 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.412315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45551 (local address 127.23.97.195:56899)
0812 11:11:12.412680 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.412698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.412827 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.413109 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.413112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.413153 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:12.413339 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.413349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.414162 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.414166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.415151 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.415163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.415848 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.415850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.415922 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.415944 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.416002 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.416056 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":12,"spinlock_wait_cycles":46720,"thread_start_us":104,"threads_started":1}
I20260812 11:11:12.416239 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.412458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:56899 (local address 127.23.97.194:45551)
0812 11:11:12.412589 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.412594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.412780 (+ 186us) server_negotiation.cc:407] Connection header received
0812 11:11:12.412937 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.412940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.412992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.413086 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:12.413457 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.414037 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.415299 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.415733 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.415856 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.415909 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.416039 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.416107 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:12.416850 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.412315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36155 (local address 127.23.97.195:41615)
0812 11:11:12.412747 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.412761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.412832 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.413109 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.413112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.413152 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:12.413344 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.413349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.414742 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.414746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.415624 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.415633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.416513 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.416517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.416619 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.416637 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.416697 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.416746 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":57,"spinlock_wait_cycles":29824,"thread_start_us":98,"threads_started":1}
I20260812 11:11:12.416833 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbc8051892c4b12ba290c03bb629a3a" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2" is_pre_election: true
I20260812 11:11:12.416939 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.412467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:41615 (local address 127.23.97.193:36155)
0812 11:11:12.412589 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.412594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.412829 (+ 235us) server_negotiation.cc:407] Connection header received
0812 11:11:12.412936 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.412939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.412992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.413086 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:12.414129 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.414627 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.415755 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.416364 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.416514 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.416571 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.416743 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.416821 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:11:12.417073 27711 raft_consensus.cc:2454] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:12.417341 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbc8051892c4b12ba290c03bb629a3a" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" is_pre_election: true
I20260812 11:11:12.417498 27476 raft_consensus.cc:2454] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:12.417577 27844 leader_election.cc:302] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:12.417874 28079 raft_consensus.cc:2781] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:12.417938 28079 raft_consensus.cc:482] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:12.417966 28079 raft_consensus.cc:3037] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.419087 28079 raft_consensus.cc:504] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.419441 28079 leader_election.cc:288] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:12.419778 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbc8051892c4b12ba290c03bb629a3a" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2"
I20260812 11:11:12.419803 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbc8051892c4b12ba290c03bb629a3a" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
I20260812 11:11:12.419881 27711 raft_consensus.cc:3037] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.419930 27476 raft_consensus.cc:3037] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.420959 27476 raft_consensus.cc:2454] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:12.420976 27711 raft_consensus.cc:2454] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:12.421285 27842 leader_election.cc:302] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 3d3ac28c47bb4feeb8059f932c10ce7a; no voters:
I20260812 11:11:12.421465 28079 raft_consensus.cc:2781] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:12.421722 28079 raft_consensus.cc:686] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [term 1 LEADER]: Becoming Leader. State: Replica: 35a043e07332486299783b33fdfbb070, State: Running, Role: LEADER
I20260812 11:11:12.422063 28079 consensus_queue.cc:237] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.424016 28114 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: TxnStatusTablet state changed. Reason: New leader 35a043e07332486299783b33fdfbb070. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.424129 28114 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:12.424422 28119 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:11:12.424620 28119 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:11:12.427129 27203 catalog_manager.cc:5495] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a043e07332486299783b33fdfbb070 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:12.470381 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.466089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.0.0.1:41364)
0812 11:11:12.466288 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.466301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.466389 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.466771 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.466774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.466783 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:12.466904 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.466909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.467909 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.467912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.468594 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.468604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.469179 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.469182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.469259 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.469304 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.470071 (+ 767us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.470089 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.470162 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.470276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 11:11:12.470371 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.466184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.23.97.195:33107)
0812 11:11:12.466344 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.466351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.466370 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:12.466512 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.466515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.466606 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.466730 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.467011 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.467804 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.468701 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.469056 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.469213 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.469440 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.469952 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.470069 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.470173 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.470228 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:11:12.476084 27476 raft_consensus.cc:1261] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.476076 27711 raft_consensus.cc:1261] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.476763 28079 consensus_queue.cc:1035] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:12.476989 28114 consensus_queue.cc:1035] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:12.480013 28083 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: TxnStatusTablet state changed. Reason: New leader 35a043e07332486299783b33fdfbb070. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.480137 28083 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.484045 28082 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: TxnStatusTablet state changed. Reason: New leader 35a043e07332486299783b33fdfbb070. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.484195 28082 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.486130 28079 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.486260 28079 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:12.486389 28079 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.486472 28079 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 35a043e07332486299783b33fdfbb070: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:12.486564 28125 mvcc.cc:205] Tried to move back new op lower bound from 7317639464854888448 to 7317639464643301376. Current Snapshot: MvccSnapshot[applied={T|T < 7317639464854888448}]
I20260812 11:11:12.488680 28083 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.488787 28083 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 771d28a868384ebe8d1d375057badaa2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.489216 28082 tablet_replica.cc:439] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } } }
I20260812 11:11:12.489331 28082 tablet_replica.cc:442] T cdbc8051892c4b12ba290c03bb629a3a P 3d3ac28c47bb4feeb8059f932c10ce7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:12.489414 28128 mvcc.cc:205] Tried to move back new op lower bound from 7317639464854888448 to 7317639464643301376. Current Snapshot: MvccSnapshot[applied={T|T < 7317639464854888448}]
I20260812 11:11:12.498276 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.494003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.23.97.194:45551)
0812 11:11:12.494156 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.494161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.494426 (+ 265us) server_negotiation.cc:407] Connection header received
0812 11:11:12.494577 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.494580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.494639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.494721 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.495036 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.495571 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.496910 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.497150 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.497238 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.497515 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.497871 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.497954 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.498068 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.498114 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:11:12.508517 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.505303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:40039 (local address 127.23.97.195:33107)
0812 11:11:12.505443 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.505447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.505584 (+ 137us) server_negotiation.cc:407] Connection header received
0812 11:11:12.505736 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.505739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.505788 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.505867 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.506104 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.506564 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.507443 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.507656 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.507767 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.508084 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.508146 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.508229 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.508325 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.508392 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:11:12.508560 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.505209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.23.97.194:40039)
0812 11:11:12.505506 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.505517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.505633 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.505882 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.505884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.505891 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:12.506005 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.506009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.506678 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.506681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.507313 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.507321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.507799 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.507805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.507888 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.507925 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.508245 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.508252 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.508314 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.508404 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260812 11:11:12.514753 27476 raft_consensus.cc:1261] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Refusing update from remote peer 771d28a868384ebe8d1d375057badaa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.514892 27954 raft_consensus.cc:1261] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Refusing update from remote peer 771d28a868384ebe8d1d375057badaa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.515192 28083 consensus_queue.cc:1035] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:12.515367 28086 consensus_queue.cc:1035] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:12.537949 28123 mvcc.cc:205] Tried to move back new op lower bound from 7317639465015996416 to 7317639463511531520. Current Snapshot: MvccSnapshot[applied={T|T < 7317639465015996416}]
I20260812 11:11:12.541455 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.535746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:44573 (local address 127.23.97.194:45551)
0812 11:11:12.535926 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.535930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.535945 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:12.536470 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.536473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.536526 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.536575 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.537933 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.538661 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.540076 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.540448 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.540529 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.540940 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.541030 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.541106 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.541267 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.541318 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 11:11:12.541496 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.535360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45551 (local address 127.23.97.195:44573)
0812 11:11:12.535852 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.535869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.535959 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.537651 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.537654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.537663 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:12.537808 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.537815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.538818 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.538824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.539930 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.539942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.540640 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.540644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.540760 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.540812 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.541149 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.541162 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.541253 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.541338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":106,"threads_started":1}
I20260812 11:11:12.566453 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:12.566654 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:12.569419 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Sending DeleteTablet for 3 replicas of tablet 622195d4e2b24993acd7676efefe9f93
I20260812 11:11:12.570158 27923 tablet_service.cc:1514] Processing DeleteTablet for tablet 622195d4e2b24993acd7676efefe9f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41362
I20260812 11:11:12.570525 27680 tablet_service.cc:1514] Processing DeleteTablet for tablet 622195d4e2b24993acd7676efefe9f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:39196
I20260812 11:11:12.570552 27440 tablet_service.cc:1514] Processing DeleteTablet for tablet 622195d4e2b24993acd7676efefe9f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:34326
I20260812 11:11:12.570979 28163 tablet_replica.cc:330] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: stopping tablet replica
I20260812 11:11:12.571344 28163 raft_consensus.cc:2229] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:12.571749 28163 raft_consensus.cc:2258] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:12.571837 27203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:12.572269 27203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:12.573478 28163 ts_tablet_manager.cc:1905] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:12.576807 28163 ts_tablet_manager.cc:1918] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:12.576972 28163 log.cc:1196] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/622195d4e2b24993acd7676efefe9f93
I20260812 11:11:12.577260 27680 tablet_service.cc:1467] Processing CreateTablet for tablet c6669e95997a4fc68ea6235cfac1d777 (DEFAULT_TABLE table=test-table [id=96d6c911c0114a17a2e8caf0c362ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:12.577497 28163 ts_tablet_manager.cc:1939] T 622195d4e2b24993acd7676efefe9f93 P 771d28a868384ebe8d1d375057badaa2: Deleting consensus metadata
I20260812 11:11:12.577554 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6669e95997a4fc68ea6235cfac1d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.578689 27183 catalog_manager.cc:4841] TS 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551): tablet 622195d4e2b24993acd7676efefe9f93 (table test-table [id=738c37eb1b3443de93fc82ace1077ab3]) successfully deleted
I20260812 11:11:12.578864 28164 tablet_replica.cc:330] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: stopping tablet replica
I20260812 11:11:12.579172 28164 raft_consensus.cc:2229] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:12.579391 28164 raft_consensus.cc:2258] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:12.579792 28071 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Bootstrap starting.
I20260812 11:11:12.580713 28071 tablet_bootstrap.cc:654] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.580857 28164 ts_tablet_manager.cc:1905] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:12.581910 27923 tablet_service.cc:1467] Processing CreateTablet for tablet c6669e95997a4fc68ea6235cfac1d777 (DEFAULT_TABLE table=test-table [id=96d6c911c0114a17a2e8caf0c362ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:12.582167 27923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6669e95997a4fc68ea6235cfac1d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.583860 28164 ts_tablet_manager.cc:1918] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:12.584000 28164 log.cc:1196] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/622195d4e2b24993acd7676efefe9f93
I20260812 11:11:12.584408 28067 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Bootstrap starting.
I20260812 11:11:12.584416 28164 ts_tablet_manager.cc:1939] T 622195d4e2b24993acd7676efefe9f93 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting consensus metadata
I20260812 11:11:12.585301 28067 tablet_bootstrap.cc:654] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.586292 27440 tablet_service.cc:1467] Processing CreateTablet for tablet c6669e95997a4fc68ea6235cfac1d777 (DEFAULT_TABLE table=test-table [id=96d6c911c0114a17a2e8caf0c362ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:12.586504 28067 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: No bootstrap required, opened a new log
I20260812 11:11:12.586515 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6669e95997a4fc68ea6235cfac1d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:12.586575 28067 ts_tablet_manager.cc:1397] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:12.586974 28067 raft_consensus.cc:348] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.587075 28067 raft_consensus.cc:374] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.587106 28067 raft_consensus.cc:729] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a043e07332486299783b33fdfbb070, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.587230 28067 consensus_queue.cc:260] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.587474 28067 ts_tablet_manager.cc:1428] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:12.588617 28069 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Bootstrap starting.
I20260812 11:11:12.589479 28069 tablet_bootstrap.cc:654] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:12.590147 27181 catalog_manager.cc:4841] TS 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155): tablet 622195d4e2b24993acd7676efefe9f93 (table test-table [id=738c37eb1b3443de93fc82ace1077ab3]) successfully deleted
I20260812 11:11:12.590677 28069 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: No bootstrap required, opened a new log
I20260812 11:11:12.590754 28069 ts_tablet_manager.cc:1397] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:12.591144 28069 raft_consensus.cc:348] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.591252 28069 raft_consensus.cc:374] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.591284 28069 raft_consensus.cc:729] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.591392 28069 consensus_queue.cc:260] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.591629 28069 ts_tablet_manager.cc:1428] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:12.592310 28071 tablet_bootstrap.cc:492] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: No bootstrap required, opened a new log
I20260812 11:11:12.592420 28071 ts_tablet_manager.cc:1397] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260812 11:11:12.592892 28071 raft_consensus.cc:348] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.593039 28071 raft_consensus.cc:374] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:12.593076 28071 raft_consensus.cc:729] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Initialized, Role: FOLLOWER
I20260812 11:11:12.593218 28071 consensus_queue.cc:260] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.593500 28071 ts_tablet_manager.cc:1428] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:12.595747 28162 tablet_replica.cc:330] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: stopping tablet replica
I20260812 11:11:12.596072 28162 raft_consensus.cc:2229] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:12.596248 28162 raft_consensus.cc:2258] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:12.598203 28162 ts_tablet_manager.cc:1905] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:12.601246 28162 ts_tablet_manager.cc:1918] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:12.601346 28162 log.cc:1196] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/622195d4e2b24993acd7676efefe9f93
I20260812 11:11:12.601647 28162 ts_tablet_manager.cc:1939] T 622195d4e2b24993acd7676efefe9f93 P 35a043e07332486299783b33fdfbb070: Deleting consensus metadata
I20260812 11:11:12.602694 27181 catalog_manager.cc:4841] TS 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107): tablet 622195d4e2b24993acd7676efefe9f93 (table test-table [id=738c37eb1b3443de93fc82ace1077ab3]) successfully deleted
I20260812 11:11:12.828599 28114 raft_consensus.cc:482] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:12.828794 28114 raft_consensus.cc:504] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.829298 28114 leader_election.cc:288] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:12.829749 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6669e95997a4fc68ea6235cfac1d777" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" is_pre_election: true
I20260812 11:11:12.829749 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6669e95997a4fc68ea6235cfac1d777" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2" is_pre_election: true
I20260812 11:11:12.829909 27711 raft_consensus.cc:2454] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:12.829909 27476 raft_consensus.cc:2454] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:12.830240 27842 leader_election.cc:302] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 3d3ac28c47bb4feeb8059f932c10ce7a; no voters:
I20260812 11:11:12.830477 28114 raft_consensus.cc:2781] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:12.830545 28114 raft_consensus.cc:482] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:12.830574 28114 raft_consensus.cc:3037] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.831601 28114 raft_consensus.cc:504] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.831941 28114 leader_election.cc:288] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:12.832280 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6669e95997a4fc68ea6235cfac1d777" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
I20260812 11:11:12.832425 27476 raft_consensus.cc:3037] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.832355 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6669e95997a4fc68ea6235cfac1d777" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2"
I20260812 11:11:12.832515 27711 raft_consensus.cc:3037] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:12.833525 27711 raft_consensus.cc:2454] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:12.833638 27476 raft_consensus.cc:2454] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:12.833849 27844 leader_election.cc:302] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:12.834026 28114 raft_consensus.cc:2781] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:12.834151 28114 raft_consensus.cc:686] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 1 LEADER]: Becoming Leader. State: Replica: 35a043e07332486299783b33fdfbb070, State: Running, Role: LEADER
I20260812 11:11:12.834306 28114 consensus_queue.cc:237] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:12.836000 27203 catalog_manager.cc:5495] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a043e07332486299783b33fdfbb070 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:12.897193 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.893740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.23.97.195:33107)
0812 11:11:12.893869 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.893873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.894072 (+ 199us) server_negotiation.cc:407] Connection header received
0812 11:11:12.894252 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.894255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.894311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.894404 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.894664 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.895169 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.896107 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.896364 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.896484 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.896737 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.896835 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.896915 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.897011 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.897063 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 11:11:12.907507 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.904127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.23.97.195:44011)
0812 11:11:12.904341 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:12.904353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:12.904473 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:12.904680 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:12.904682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:12.904687 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:12.904760 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.904764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.905726 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.905729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.906317 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:12.906326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.906894 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.906897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:12.906961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.906986 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.907255 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.907260 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:11:12.907310 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 11:11:12.907380 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260812 11:11:12.907516 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:12.904230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:44011 (local address 127.23.97.195:33107)
0812 11:11:12.904340 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:12.904344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:12.904444 (+ 100us) server_negotiation.cc:407] Connection header received
0812 11:11:12.904541 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:12.904543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:12.904584 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:12.904659 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:12.904852 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.905595 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.906431 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:12.906766 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:12.906859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:12.907100 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:12.907170 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:12.907236 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:12.907318 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 11:11:12.907361 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:11:12.913004 27711 raft_consensus.cc:1261] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.913087 27476 raft_consensus.cc:1261] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:12.913412 28114 consensus_queue.cc:1035] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:12.913592 28131 consensus_queue.cc:1035] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:11:12.926957 28123 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:11:14.479391 27202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:14.479542 27202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:14.481856 27202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Sending DeleteTablet for 3 replicas of tablet c6669e95997a4fc68ea6235cfac1d777
I20260812 11:11:14.482491 27922 tablet_service.cc:1514] Processing DeleteTablet for tablet c6669e95997a4fc68ea6235cfac1d777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41362
I20260812 11:11:14.482491 27680 tablet_service.cc:1514] Processing DeleteTablet for tablet c6669e95997a4fc68ea6235cfac1d777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:39196
I20260812 11:11:14.482515 27440 tablet_service.cc:1514] Processing DeleteTablet for tablet c6669e95997a4fc68ea6235cfac1d777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:34326
I20260812 11:11:14.482894 28203 tablet_replica.cc:330] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: stopping tablet replica
I20260812 11:11:14.483093 28203 raft_consensus.cc:2229] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:14.483443 28203 raft_consensus.cc:2258] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.483724 28204 tablet_replica.cc:330] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: stopping tablet replica
I20260812 11:11:14.483881 28204 raft_consensus.cc:2229] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:14.484002 28204 raft_consensus.cc:2258] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.484251 28209 tablet_replica.cc:330] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: stopping tablet replica
I20260812 11:11:14.484431 28209 raft_consensus.cc:2229] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:14.484601 28209 raft_consensus.cc:2258] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.485299 28203 ts_tablet_manager.cc:1905] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.487071 28204 ts_tablet_manager.cc:1905] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.487840 28209 ts_tablet_manager.cc:1905] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.488140 28203 ts_tablet_manager.cc:1918] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:11:14.488278 28203 log.cc:1196] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/c6669e95997a4fc68ea6235cfac1d777
I20260812 11:11:14.488655 28203 ts_tablet_manager.cc:1939] T c6669e95997a4fc68ea6235cfac1d777 P 35a043e07332486299783b33fdfbb070: Deleting consensus metadata
I20260812 11:11:14.489392 27181 catalog_manager.cc:4841] TS 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107): tablet c6669e95997a4fc68ea6235cfac1d777 (table test-table [id=96d6c911c0114a17a2e8caf0c362ed50]) successfully deleted
I20260812 11:11:14.490326 28204 ts_tablet_manager.cc:1918] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:11:14.490427 28204 log.cc:1196] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/c6669e95997a4fc68ea6235cfac1d777
I20260812 11:11:14.490703 28204 ts_tablet_manager.cc:1939] T c6669e95997a4fc68ea6235cfac1d777 P 771d28a868384ebe8d1d375057badaa2: Deleting consensus metadata
I20260812 11:11:14.491328 27183 catalog_manager.cc:4841] TS 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551): tablet c6669e95997a4fc68ea6235cfac1d777 (table test-table [id=96d6c911c0114a17a2e8caf0c362ed50]) successfully deleted
I20260812 11:11:14.491638 28209 ts_tablet_manager.cc:1918] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:11:14.491734 28209 log.cc:1196] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/c6669e95997a4fc68ea6235cfac1d777
I20260812 11:11:14.492033 28209 ts_tablet_manager.cc:1939] T c6669e95997a4fc68ea6235cfac1d777 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting consensus metadata
I20260812 11:11:14.492697 27181 catalog_manager.cc:4841] TS 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155): tablet c6669e95997a4fc68ea6235cfac1d777 (table test-table [id=96d6c911c0114a17a2e8caf0c362ed50]) successfully deleted
Aug 12 11:11:14 dist-test-slave-zpvx krb5kdc[27130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:11:14 dist-test-slave-zpvx krb5kdc[27130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:14.516496 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:14.503294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.23.97.254:45609)
0812 11:11:14.503588 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:14.503594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:14.503608 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:14.503732 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:14.503734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:14.503777 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:14.503853 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:14.504546 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:14.505099 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:14.505879 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:14.506105 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:14.506184 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:14.513380 (+ 7196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:14.513383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:14.513386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:14.513410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:14.514950 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:14.515316 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:14.515319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:14.515320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:14.515397 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:14.515625 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:14.515627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:14.515628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:14.515803 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:14.515904 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:14.516100 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 11:11:14.516198 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260812 11:11:14.518922 27202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:14.519361 27202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:14.523921 27440 tablet_service.cc:1467] Processing CreateTablet for tablet 2054ec7601e4401b9327cf401be506e2 (DEFAULT_TABLE table=test-table [id=952c1604190849f3b59eb13510f27c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.524000 27680 tablet_service.cc:1467] Processing CreateTablet for tablet 2054ec7601e4401b9327cf401be506e2 (DEFAULT_TABLE table=test-table [id=952c1604190849f3b59eb13510f27c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.524257 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054ec7601e4401b9327cf401be506e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.524261 27922 tablet_service.cc:1467] Processing CreateTablet for tablet 2054ec7601e4401b9327cf401be506e2 (DEFAULT_TABLE table=test-table [id=952c1604190849f3b59eb13510f27c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.524325 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054ec7601e4401b9327cf401be506e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.524583 27922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054ec7601e4401b9327cf401be506e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.526842 28224 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Bootstrap starting.
I20260812 11:11:14.527222 28225 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Bootstrap starting.
I20260812 11:11:14.527305 28227 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Bootstrap starting.
I20260812 11:11:14.527763 28224 tablet_bootstrap.cc:654] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.528299 28227 tablet_bootstrap.cc:654] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.528627 28225 tablet_bootstrap.cc:654] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.529238 28224 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: No bootstrap required, opened a new log
I20260812 11:11:14.529554 28227 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: No bootstrap required, opened a new log
I20260812 11:11:14.529553 28224 ts_tablet_manager.cc:1397] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:14.529639 28227 ts_tablet_manager.cc:1397] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:14.529966 28224 raft_consensus.cc:348] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.529981 28227 raft_consensus.cc:348] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.530093 28224 raft_consensus.cc:374] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.530092 28227 raft_consensus.cc:374] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.530128 28224 raft_consensus.cc:729] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.530128 28227 raft_consensus.cc:729] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a043e07332486299783b33fdfbb070, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.530260 28224 consensus_queue.cc:260] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.530260 28227 consensus_queue.cc:260] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.530635 28227 ts_tablet_manager.cc:1428] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:14.530635 28224 ts_tablet_manager.cc:1428] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:14.531782 28225 tablet_bootstrap.cc:492] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: No bootstrap required, opened a new log
I20260812 11:11:14.531939 28225 ts_tablet_manager.cc:1397] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:11:14.532339 28225 raft_consensus.cc:348] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.532568 28225 raft_consensus.cc:374] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.532661 28225 raft_consensus.cc:729] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.532876 28225 consensus_queue.cc:260] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.533308 28225 ts_tablet_manager.cc:1428] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:14.637804 28131 raft_consensus.cc:482] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:14.637945 28131 raft_consensus.cc:504] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.638494 28131 leader_election.cc:288] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:14.638999 27474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054ec7601e4401b9327cf401be506e2" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" is_pre_election: true
I20260812 11:11:14.639014 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054ec7601e4401b9327cf401be506e2" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2" is_pre_election: true
I20260812 11:11:14.639175 27474 raft_consensus.cc:2454] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:14.639189 27711 raft_consensus.cc:2454] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 0.
I20260812 11:11:14.639576 27844 leader_election.cc:302] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:14.639807 28131 raft_consensus.cc:2781] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:14.639889 28131 raft_consensus.cc:482] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:14.639948 28131 raft_consensus.cc:3037] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:14.641062 28131 raft_consensus.cc:504] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.641456 28131 leader_election.cc:288] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155), 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551)
I20260812 11:11:14.641808 27474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054ec7601e4401b9327cf401be506e2" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a"
I20260812 11:11:14.641870 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054ec7601e4401b9327cf401be506e2" candidate_uuid: "35a043e07332486299783b33fdfbb070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2"
I20260812 11:11:14.641955 27474 raft_consensus.cc:3037] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:14.642038 27711 raft_consensus.cc:3037] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:14.643077 27474 raft_consensus.cc:2454] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:14.643081 27711 raft_consensus.cc:2454] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a043e07332486299783b33fdfbb070 in term 1.
I20260812 11:11:14.643399 27842 leader_election.cc:302] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 3d3ac28c47bb4feeb8059f932c10ce7a; no voters:
I20260812 11:11:14.643561 28131 raft_consensus.cc:2781] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:14.643661 28131 raft_consensus.cc:686] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 1 LEADER]: Becoming Leader. State: Replica: 35a043e07332486299783b33fdfbb070, State: Running, Role: LEADER
I20260812 11:11:14.643829 28131 consensus_queue.cc:237] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.645821 27201 catalog_manager.cc:5495] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a043e07332486299783b33fdfbb070 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "35a043e07332486299783b33fdfbb070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:14.685590 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:14.682360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.23.97.195:33107)
0812 11:11:14.682628 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:14.682635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:14.682646 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:14.682684 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:14.682686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:14.682728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:14.682797 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:14.683030 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:14.683553 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:14.684423 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:14.684670 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:14.684860 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:14.685121 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:14.685200 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:14.685291 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:14.685385 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 11:11:14.685434 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":131,"threads_started":1}
I20260812 11:11:14.698312 27711 raft_consensus.cc:1261] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:14.698711 27474 raft_consensus.cc:1261] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Refusing update from remote peer 35a043e07332486299783b33fdfbb070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:14.698784 28131 consensus_queue.cc:1035] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:14.699350 28131 consensus_queue.cc:1035] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:14.731134 27201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:14.731283 27201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:14.733305 27201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Sending DeleteTablet for 3 replicas of tablet 2054ec7601e4401b9327cf401be506e2
I20260812 11:11:14.733840 27440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2054ec7601e4401b9327cf401be506e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:34326
I20260812 11:11:14.733911 27680 tablet_service.cc:1514] Processing DeleteTablet for tablet 2054ec7601e4401b9327cf401be506e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:39196
I20260812 11:11:14.734035 28209 tablet_replica.cc:330] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: stopping tablet replica
I20260812 11:11:14.734104 28204 tablet_replica.cc:330] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: stopping tablet replica
I20260812 11:11:14.734098 27922 tablet_service.cc:1514] Processing DeleteTablet for tablet 2054ec7601e4401b9327cf401be506e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:14 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41362
I20260812 11:11:14.734150 28209 raft_consensus.cc:2229] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:14.734217 28204 raft_consensus.cc:2229] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:14.734265 28209 raft_consensus.cc:2258] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.734328 28204 raft_consensus.cc:2258] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.734503 28203 tablet_replica.cc:330] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: stopping tablet replica
I20260812 11:11:14.734606 28203 raft_consensus.cc:2229] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:14.734735 28203 raft_consensus.cc:2258] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:14.734761 27201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:14.735160 27201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:14.735469 28203 ts_tablet_manager.cc:1905] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.735526 28204 ts_tablet_manager.cc:1905] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.738003 28203 ts_tablet_manager.cc:1918] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:14.738003 28204 ts_tablet_manager.cc:1918] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:14.738085 28204 log.cc:1196] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/2054ec7601e4401b9327cf401be506e2
I20260812 11:11:14.738205 28203 log.cc:1196] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/2054ec7601e4401b9327cf401be506e2
I20260812 11:11:14.738368 28204 ts_tablet_manager.cc:1939] T 2054ec7601e4401b9327cf401be506e2 P 771d28a868384ebe8d1d375057badaa2: Deleting consensus metadata
I20260812 11:11:14.738498 28203 ts_tablet_manager.cc:1939] T 2054ec7601e4401b9327cf401be506e2 P 35a043e07332486299783b33fdfbb070: Deleting consensus metadata
I20260812 11:11:14.739037 27183 catalog_manager.cc:4841] TS 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551): tablet 2054ec7601e4401b9327cf401be506e2 (table test-table [id=952c1604190849f3b59eb13510f27c17]) successfully deleted
I20260812 11:11:14.739332 27181 catalog_manager.cc:4841] TS 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107): tablet 2054ec7601e4401b9327cf401be506e2 (table test-table [id=952c1604190849f3b59eb13510f27c17]) successfully deleted
I20260812 11:11:14.739430 28209 ts_tablet_manager.cc:1905] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:14.739581 27440 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac77f75ae044e9bbc39cf394bcfff87 (DEFAULT_TABLE table=test-table [id=824a83939331458294b338823124dc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.739845 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac77f75ae044e9bbc39cf394bcfff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.739883 27922 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac77f75ae044e9bbc39cf394bcfff87 (DEFAULT_TABLE table=test-table [id=824a83939331458294b338823124dc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.740093 27922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac77f75ae044e9bbc39cf394bcfff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.741712 28225 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a: Bootstrap starting.
I20260812 11:11:14.742053 28209 ts_tablet_manager.cc:1918] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:14.742333 27680 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac77f75ae044e9bbc39cf394bcfff87 (DEFAULT_TABLE table=test-table [id=824a83939331458294b338823124dc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:14.742411 28227 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Bootstrap starting.
I20260812 11:11:14.742568 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac77f75ae044e9bbc39cf394bcfff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:14.743165 28227 tablet_bootstrap.cc:654] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.744275 28227 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: No bootstrap required, opened a new log
I20260812 11:11:14.744349 28227 ts_tablet_manager.cc:1397] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:14.744835 28224 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Bootstrap starting.
I20260812 11:11:14.744777 28227 raft_consensus.cc:348] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.744884 28227 raft_consensus.cc:374] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.744921 28227 raft_consensus.cc:729] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a043e07332486299783b33fdfbb070, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.742720 28225 tablet_bootstrap.cc:654] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.745038 28227 consensus_queue.cc:260] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.745334 28227 ts_tablet_manager.cc:1428] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:14.745661 28224 tablet_bootstrap.cc:654] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:14.745671 28209 log.cc:1196] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/2054ec7601e4401b9327cf401be506e2
I20260812 11:11:14.745951 28209 ts_tablet_manager.cc:1939] T 2054ec7601e4401b9327cf401be506e2 P 3d3ac28c47bb4feeb8059f932c10ce7a: Deleting consensus metadata
I20260812 11:11:14.747187 28225 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a: No bootstrap required, opened a new log
I20260812 11:11:14.747293 28225 ts_tablet_manager.cc:1397] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 11:11:14.747359 28224 tablet_bootstrap.cc:492] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: No bootstrap required, opened a new log
I20260812 11:11:14.747473 28224 ts_tablet_manager.cc:1397] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 11:11:14.747756 28225 raft_consensus.cc:348] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.747872 28225 raft_consensus.cc:374] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.747828 28224 raft_consensus.cc:348] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.747905 28225 raft_consensus.cc:729] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.747929 28224 raft_consensus.cc:374] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:14.747963 28224 raft_consensus.cc:729] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d28a868384ebe8d1d375057badaa2, State: Initialized, Role: FOLLOWER
I20260812 11:11:14.748037 28225 consensus_queue.cc:260] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.748080 28224 consensus_queue.cc:260] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:14.748310 28225 ts_tablet_manager.cc:1428] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:14.748337 28224 ts_tablet_manager.cc:1428] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:14.748943 27181 catalog_manager.cc:4841] TS 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155): tablet 2054ec7601e4401b9327cf401be506e2 (table test-table [id=952c1604190849f3b59eb13510f27c17]) successfully deleted
I20260812 11:11:15.004030 28234 raft_consensus.cc:482] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:15.004242 28234 raft_consensus.cc:504] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:15.005419 28234 leader_election.cc:288] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551), 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107)
I20260812 11:11:15.011292 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.005999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:59201 (local address 127.23.97.195:33107)
0812 11:11:15.006156 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:15.006163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:15.006180 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:11:15.006529 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:15.006533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:15.006601 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:15.006703 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:15.007188 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.008048 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.009972 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.010601 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.010729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.010788 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:15.011076 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:11:15.011157 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:11:15.011296 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.005855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:47441 (local address 127.23.97.194:45551)
0812 11:11:15.006170 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:15.006181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:15.006536 (+ 355us) server_negotiation.cc:407] Connection header received
0812 11:11:15.006598 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:15.006601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:15.006657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:15.006748 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:15.007306 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.008086 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.009972 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.010597 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.010730 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.010793 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:15.011076 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 11:11:15.011156 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260812 11:11:15.011389 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.005684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45551 (local address 127.23.97.193:47441)
0812 11:11:15.006338 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:15.006353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:15.006438 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:15.006763 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:15.006766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:15.006777 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:15.006994 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.007000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.008222 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.008225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.009827 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.009841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.010748 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.010751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.010910 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.010959 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:11:15.011041 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:11:15.011199 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":391,"thread_start_us":100,"threads_started":1}
I20260812 11:11:15.011649 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.005684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.23.97.193:59201)
0812 11:11:15.006093 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:15.006110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:15.006253 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:15.006744 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:15.006750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:15.006770 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:15.006989 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.006998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.008192 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.008198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.009827 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.009841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.010733 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.010739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.010879 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.010962 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 11:11:15.011041 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:11:15.011200 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":194,"spinlock_wait_cycles":31360,"thread_start_us":84,"threads_started":1}
I20260812 11:11:15.011935 27709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac77f75ae044e9bbc39cf394bcfff87" candidate_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2" is_pre_election: true
I20260812 11:11:15.012101 27709 raft_consensus.cc:2454] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3ac28c47bb4feeb8059f932c10ce7a in term 0.
I20260812 11:11:15.012519 27365 leader_election.cc:302] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3ac28c47bb4feeb8059f932c10ce7a, 771d28a868384ebe8d1d375057badaa2; no voters:
I20260812 11:11:15.012543 27954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac77f75ae044e9bbc39cf394bcfff87" candidate_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a043e07332486299783b33fdfbb070" is_pre_election: true
I20260812 11:11:15.012692 27954 raft_consensus.cc:2454] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3ac28c47bb4feeb8059f932c10ce7a in term 0.
I20260812 11:11:15.012821 28234 raft_consensus.cc:2781] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:15.012909 28234 raft_consensus.cc:482] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:15.012967 28234 raft_consensus.cc:3037] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:15.014017 28234 raft_consensus.cc:504] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:15.014614 28234 leader_election.cc:288] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [CANDIDATE]: Term 1 election: Requested vote from peers 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551), 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107)
I20260812 11:11:15.015021 27709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac77f75ae044e9bbc39cf394bcfff87" candidate_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771d28a868384ebe8d1d375057badaa2"
I20260812 11:11:15.015029 27954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac77f75ae044e9bbc39cf394bcfff87" candidate_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a043e07332486299783b33fdfbb070"
I20260812 11:11:15.015194 27709 raft_consensus.cc:3037] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:15.015199 27954 raft_consensus.cc:3037] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:15.016179 27954 raft_consensus.cc:2454] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3ac28c47bb4feeb8059f932c10ce7a in term 1.
I20260812 11:11:15.016198 27709 raft_consensus.cc:2454] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3ac28c47bb4feeb8059f932c10ce7a in term 1.
I20260812 11:11:15.016496 27362 leader_election.cc:302] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a043e07332486299783b33fdfbb070, 3d3ac28c47bb4feeb8059f932c10ce7a; no voters:
I20260812 11:11:15.016672 28234 raft_consensus.cc:2781] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:15.016947 28234 raft_consensus.cc:686] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [term 1 LEADER]: Becoming Leader. State: Replica: 3d3ac28c47bb4feeb8059f932c10ce7a, State: Running, Role: LEADER
I20260812 11:11:15.017206 28234 consensus_queue.cc:237] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } }
I20260812 11:11:15.019601 27202 catalog_manager.cc:5495] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193). New cstate: current_term: 1 leader_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ac28c47bb4feeb8059f932c10ce7a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:15.045702 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.042285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.23.97.193:36155)
0812 11:11:15.042593 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:15.042600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:15.042611 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:15.042678 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:15.042680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:15.042732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:15.042832 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:15.043017 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.043543 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.044307 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.044540 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.044616 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.044843 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:15.045319 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:15.045406 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:15.045475 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 11:11:15.045519 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":147,"threads_started":1}
I20260812 11:11:15.056198 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.052415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:42491 (local address 127.23.97.195:33107)
0812 11:11:15.052552 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:15.052557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:15.052700 (+ 143us) server_negotiation.cc:407] Connection header received
0812 11:11:15.052857 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:15.052860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:15.052914 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:15.052993 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:15.053242 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.053727 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.054662 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.055018 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.055124 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.055536 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:15.055661 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:15.055741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:15.055948 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 11:11:15.056009 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:15.056221 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.052259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:33107 (local address 127.23.97.193:42491)
0812 11:11:15.052618 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:15.052631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:15.052748 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:15.053005 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:15.053008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:15.053016 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:15.053131 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.053137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.053848 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.053853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.054499 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.054531 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.055165 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.055170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.055315 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.055370 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:15.055787 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:15.055796 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:11:15.055926 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 11:11:15.056009 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":138,"threads_started":1}
I20260812 11:11:15.065471 27709 raft_consensus.cc:1261] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Refusing update from remote peer 3d3ac28c47bb4feeb8059f932c10ce7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:15.065482 27954 raft_consensus.cc:1261] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Refusing update from remote peer 3d3ac28c47bb4feeb8059f932c10ce7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:15.066071 28257 consensus_queue.cc:1035] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a043e07332486299783b33fdfbb070" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:15.066459 28234 consensus_queue.cc:1035] T 1ac77f75ae044e9bbc39cf394bcfff87 P 3d3ac28c47bb4feeb8059f932c10ce7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "771d28a868384ebe8d1d375057badaa2" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:15.082928 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.079473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49429 (local address 127.23.97.193:36155)
0812 11:11:15.079573 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:15.079578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:15.079794 (+ 216us) server_negotiation.cc:407] Connection header received
0812 11:11:15.079929 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:15.079931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:15.079972 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:15.080026 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:15.080219 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.080777 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.081875 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.082199 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.082314 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.082584 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:15.082648 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:15.082704 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:15.082791 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 11:11:15.082832 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 11:11:15.082967 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:15.079425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36155 (local address 127.23.97.195:49429)
0812 11:11:15.079698 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:15.079717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:15.079816 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:15.080052 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:15.080054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:15.080061 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:15.080147 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.080152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.080896 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.080899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.081761 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:15.081772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:15.082330 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:15.082336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:15.082432 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:15.082465 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:15.082727 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:15.082733 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:11:15.082786 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 11:11:15.082851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":118,"threads_started":1}
W20260812 11:11:15.083752 28183 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:11:16.639273 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:16.639430 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:16.641783 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e049c9d8ea3744d9aa9457047c11f185: Sending DeleteTablet for 3 replicas of tablet 1ac77f75ae044e9bbc39cf394bcfff87
I20260812 11:11:16.642436 27680 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac77f75ae044e9bbc39cf394bcfff87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:16 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:39196
I20260812 11:11:16.642480 27922 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac77f75ae044e9bbc39cf394bcfff87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:16 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41362
I20260812 11:11:16.642772 28309 tablet_replica.cc:330] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: stopping tablet replica
I20260812 11:11:16.642831 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27333
I20260812 11:11:16.653155 28312 tablet_replica.cc:330] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: stopping tablet replica
I20260812 11:11:16.653316 28312 raft_consensus.cc:2229] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:16.653445 28312 raft_consensus.cc:2258] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:16.653940 28309 raft_consensus.cc:2229] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:16.654075 28309 raft_consensus.cc:2258] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:16.654201 28312 ts_tablet_manager.cc:1905] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:16.655438 28309 ts_tablet_manager.cc:1905] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:16.657205 28312 ts_tablet_manager.cc:1918] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:11:16.657335 28312 log.cc:1196] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/1ac77f75ae044e9bbc39cf394bcfff87
I20260812 11:11:16.657661 28312 ts_tablet_manager.cc:1939] T 1ac77f75ae044e9bbc39cf394bcfff87 P 35a043e07332486299783b33fdfbb070: Deleting consensus metadata
I20260812 11:11:16.658394 28309 ts_tablet_manager.cc:1918] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:11:16.658504 28309 log.cc:1196] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/1ac77f75ae044e9bbc39cf394bcfff87
I20260812 11:11:16.658499 27181 catalog_manager.cc:4841] TS 35a043e07332486299783b33fdfbb070 (127.23.97.195:33107): tablet 1ac77f75ae044e9bbc39cf394bcfff87 (table test-table [id=824a83939331458294b338823124dc38]) successfully deleted
I20260812 11:11:16.658823 28309 ts_tablet_manager.cc:1939] T 1ac77f75ae044e9bbc39cf394bcfff87 P 771d28a868384ebe8d1d375057badaa2: Deleting consensus metadata
I20260812 11:11:16.659482 27183 catalog_manager.cc:4841] TS 771d28a868384ebe8d1d375057badaa2 (127.23.97.194:45551): tablet 1ac77f75ae044e9bbc39cf394bcfff87 (table test-table [id=824a83939331458294b338823124dc38]) successfully deleted
W20260812 11:11:16.666116 27181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:36155 (error 108)
I20260812 11:11:16.666329 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27576
I20260812 11:11:16.671664 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:16.671108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36155 (local address 127.0.0.1:54846)
0812 11:11:16.671424 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:16.671522 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:36155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
W20260812 11:11:16.671926 27181 catalog_manager.cc:4584] TS 3d3ac28c47bb4feeb8059f932c10ce7a (127.23.97.193:36155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ac77f75ae044e9bbc39cf394bcfff87: Network error: Client connection negotiation failed: client connection to 127.23.97.193:36155: connect: Connection refused (error 111)
I20260812 11:11:16.680307 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27815
I20260812 11:11:16.692209 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27149
2026-08-12T11:11:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7187 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:16 dist-test-slave-zpvx krb5kdc[28320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:16 dist-test-slave-zpvx krb5kdc[28320](info): set up 2 sockets
Aug 12 11:11:16 dist-test-slave-zpvx krb5kdc[28320](info): commencing operation
krb5kdc: starting...
W20260812 11:11:18.040112 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:18.164762 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36269
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:36269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:18.275679 28347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:18.275921 28347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:18.275959 28347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:18.280578 28347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:18.280630 28347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:18.280649 28347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:18.280666 28347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:18.280681 28347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:18.285892 28347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:36269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:18.286826 28347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:18.287750 28347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:18.296484 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.296545 28347 server_base.cc:1034] running on GCE node
W20260812 11:11:18.296484 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:18.296550 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.297353 28347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:18.298381 28347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:18.299573 28347 hybrid_clock.cc:648] HybridClock initialized: now 1786533078299555 us; error 31 us; skew 500 ppm
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:18.302349 28347 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:18.303175 28347 webserver.cc:460] Webserver started at http://127.23.97.254:41983/ using document root <none> and password file <none>
I20260812 11:11:18.303727 28347 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:18.303820 28347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:18.304050 28347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:18.306083 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "fcae57bebaa64de1962fd62298a70df0"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "82d36c176ac139aa06a7002a00ba30bf"
server_key_iv: "c17103e9ddd6726e6d559843aeb9c9de"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.306660 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "fcae57bebaa64de1962fd62298a70df0"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "82d36c176ac139aa06a7002a00ba30bf"
server_key_iv: "c17103e9ddd6726e6d559843aeb9c9de"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.310343 28347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:18.312709 28375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.313939 28347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:18.314082 28347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "fcae57bebaa64de1962fd62298a70df0"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "82d36c176ac139aa06a7002a00ba30bf"
server_key_iv: "c17103e9ddd6726e6d559843aeb9c9de"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.314199 28347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:18.327195 28347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:18.330350 28347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:18.330555 28347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:18.338045 28347 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:36269
I20260812 11:11:18.338071 28468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:36269 every 8 connection(s)
I20260812 11:11:18.338980 28347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:18.340652 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28347
I20260812 11:11:18.340767 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:18.341059 23943 external_mini_cluster.cc:1428] Setting key a8f9463d40eb13802c8d2a002a901a95
I20260812 11:11:18.342136 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:18.347721 28470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0: Bootstrap starting.
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:18.350275 28470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:18.351236 28470 log.cc:824] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:18.353606 28470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0: No bootstrap required, opened a new log
I20260812 11:11:18.355667 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:18.342500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.23.97.254:36269)
0812 11:11:18.342968 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:18.342984 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:18.343019 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:18.343664 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:18.343680 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:18.344005 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:18.344350 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:18.345194 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.346180 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.347051 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.347516 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.347704 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:18.350489 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:18.350498 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:18.350519 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:18.350558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:18.352563 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.353008 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.353014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.353020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.353157 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.353426 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.353429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.353431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.353944 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:18.354138 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:18.354418 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 11:11:18.354717 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":150,"threads_started":1}
I20260812 11:11:18.358029 28470 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } }
I20260812 11:11:18.358299 28470 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:18.358340 28470 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcae57bebaa64de1962fd62298a70df0, State: Initialized, Role: FOLLOWER
I20260812 11:11:18.358747 28470 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } }
I20260812 11:11:18.358871 28470 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:18.358917 28470 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:18.358986 28470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:18.359999 28470 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } }
I20260812 11:11:18.360294 28470 leader_election.cc:302] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcae57bebaa64de1962fd62298a70df0; no voters:
I20260812 11:11:18.360598 28470 leader_election.cc:288] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:18.360737 28482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:18.360989 28482 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [term 1 LEADER]: Becoming Leader. State: Replica: fcae57bebaa64de1962fd62298a70df0, State: Running, Role: LEADER
I20260812 11:11:18.361351 28482 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } }
I20260812 11:11:18.361788 28470 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:18.362951 28485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcae57bebaa64de1962fd62298a70df0. Latest consensus state: current_term: 1 leader_uuid: "fcae57bebaa64de1962fd62298a70df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } } }
I20260812 11:11:18.363090 28485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:18.363531 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcae57bebaa64de1962fd62298a70df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae57bebaa64de1962fd62298a70df0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36269 } } }
I20260812 11:11:18.363570 28496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:18.363696 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:18.366540 28496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:18.372722 28496 catalog_manager.cc:1347] Generated new cluster ID: f02a493afd634fed9f65e88b23d9198e
I20260812 11:11:18.372792 28496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:18.384413 28496 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:18.385624 28496 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:18.399544 28496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcae57bebaa64de1962fd62298a70df0: Generated new TSK 0
I20260812 11:11:18.400434 28496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:18.439855 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:18.551702 28528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:18.551945 28528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:18.551987 28528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:18.556082 28528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:18.556141 28528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:18.556222 28528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:18.562013 28528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:18.563271 28528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:18.564388 28528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:18.573808 28540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:18.573817 28537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.573997 28528 server_base.cc:1034] running on GCE node
W20260812 11:11:18.574172 28535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.574591 28528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:18.575287 28528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:18.576439 28528 hybrid_clock.cc:648] HybridClock initialized: now 1786533078576405 us; error 43 us; skew 500 ppm
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:18.579103 28528 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:18.579886 28528 webserver.cc:460] Webserver started at http://127.23.97.193:35363/ using document root <none> and password file <none>
I20260812 11:11:18.580443 28528 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:18.580525 28528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:18.580693 28528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:18.582612 28528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "4f65cb1285d84e39a58e2ce734ee2b17"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "022787dd4b4221ae91e93d6609f2b1a3"
server_key_iv: "8280fdccdabca19e11d7304d9b4893bd"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.583119 28528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "4f65cb1285d84e39a58e2ce734ee2b17"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "022787dd4b4221ae91e93d6609f2b1a3"
server_key_iv: "8280fdccdabca19e11d7304d9b4893bd"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.586651 28528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:18.588963 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.590132 28528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:18.590255 28528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "4f65cb1285d84e39a58e2ce734ee2b17"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "022787dd4b4221ae91e93d6609f2b1a3"
server_key_iv: "8280fdccdabca19e11d7304d9b4893bd"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.590359 28528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:18.607491 28528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:18.610496 28528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:18.610698 28528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:18.611284 28528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:18.612110 28528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:18.612157 28528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.612228 28528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:18.612262 28528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.621483 28528 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:46721
I20260812 11:11:18.621526 28761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:46721 every 8 connection(s)
I20260812 11:11:18.622346 28528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:18.626091 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28528
I20260812 11:11:18.626240 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:18.626566 23943 external_mini_cluster.cc:1428] Setting key 280dadf761680b84bbc3174c23d89b89
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:18.636134 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:18.624109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:36755 (local address 127.23.97.254:36269)
0812 11:11:18.624282 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:18.624287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:18.625339 (+ 1052us) server_negotiation.cc:407] Connection header received
0812 11:11:18.626187 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:18.626191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:18.626259 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:18.626359 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:18.627635 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.628218 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.629416 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.629688 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.629808 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:18.632250 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:18.632257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:18.632263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:18.632297 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:18.634059 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.634574 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.634581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.634586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.634679 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.635015 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.635021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.635023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.635262 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:18.635375 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:18.635813 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 11:11:18.635947 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:11:18.636829 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:18.624465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36269 (local address 127.23.97.193:36755)
0812 11:11:18.625192 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:18.625235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:18.625970 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:18.626538 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:18.626545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:18.626944 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:18.627494 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:18.627508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.628357 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.628404 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:18.629289 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:18.629299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.629875 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.629881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:18.630035 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:18.630617 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:18.630633 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:18.632067 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:18.634182 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:18.634186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:18.634197 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:18.634466 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:18.634779 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:18.634782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:18.634783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:18.634911 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:18.635367 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:18.635372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:18.635578 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 11:11:18.635726 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 11:11:18.635956 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:18.638028 28765 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36269
I20260812 11:11:18.638275 28765 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:18.638700 28765 heartbeater.cc:511] Master 127.23.97.254:36269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:18.640144 28401 ts_manager.cc:194] Registered new tserver with Master: 4f65cb1285d84e39a58e2ce734ee2b17 (127.23.97.193:46721)
I20260812 11:11:18.641852 28401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:36755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:18.655521 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:18.770764 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:18.771018 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:18.771086 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:18.775063 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:18.775143 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:18.775229 28775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:18.780198 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:18.781158 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:18.782065 28775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:18.791431 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.791486 28775 server_base.cc:1034] running on GCE node
W20260812 11:11:18.791421 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:18.791402 28786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:18.792043 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:18.792738 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:18.793916 28775 hybrid_clock.cc:648] HybridClock initialized: now 1786533078793893 us; error 47 us; skew 500 ppm
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:18.796739 28775 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:18.797533 28775 webserver.cc:460] Webserver started at http://127.23.97.194:36927/ using document root <none> and password file <none>
I20260812 11:11:18.798061 28775 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:18.798154 28775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:18.798405 28775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:18.800318 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "5b0158b6e47e411fad638cb0c37465a9"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "81994f9d57d85d46f2d692103ddf5340"
server_key_iv: "c530a9328c8bdf52d5bd01c15304d7ca"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.800982 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "5b0158b6e47e411fad638cb0c37465a9"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "81994f9d57d85d46f2d692103ddf5340"
server_key_iv: "c530a9328c8bdf52d5bd01c15304d7ca"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.804646 28775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:11:18.806954 28797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.808065 28775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:18.808199 28775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "5b0158b6e47e411fad638cb0c37465a9"
format_stamp: "Formatted at 2026-08-12 11:11:18 on dist-test-slave-zpvx"
server_key: "81994f9d57d85d46f2d692103ddf5340"
server_key_iv: "c530a9328c8bdf52d5bd01c15304d7ca"
server_key_version: "encryptionkey@0"
I20260812 11:11:18.808331 28775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:18.819448 28775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:18.822528 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:18.822732 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:18.823269 28775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:18.824066 28775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:18.824138 28775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.824210 28775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:18.824262 28775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:18.832281 28775 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:44945
I20260812 11:11:18.832348 29006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:44945 every 8 connection(s)
I20260812 11:11:18.833225 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:18.841593 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28775
I20260812 11:11:18.841691 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
Aug 12 11:11:18 dist-test-slave-zpvx krb5kdc[28320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:18.842000 23943 external_mini_cluster.cc:1428] Setting key abb365b77df2776cd8fcb83a17f5796a
I20260812 11:11:18.846274 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:18.834948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:40509 (local address 127.23.97.254:36269)
0812 11:11:18.835091 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:18.835094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:18.836040 (+ 946us) server_negotiation.cc:407] Connection header received
0812 11:11:18.836941 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:18.836943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:18.836990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:18.837063 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:18.838132 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.838647 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.839738 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.839947 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.840022 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:18.842632 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:18.842636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:18.842638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:18.842663 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:18.844103 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.844766 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.844768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.844770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.844842 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:18.845232 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:18.845235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:18.845236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:18.845389 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:18.845499 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:18.845994 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 11:11:18.846109 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:11:18.847083 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:18.835277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36269 (local address 127.23.97.194:40509)
0812 11:11:18.835904 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:18.835942 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:18.836762 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:18.837220 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:18.837226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:18.837548 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:18.838001 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:18.838014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.838778 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.838780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:18.839628 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:18.839637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:18.840070 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:18.840072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:18.840216 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:18.840842 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:18.840857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:18.842422 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:18.844231 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:18.844240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:18.844257 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:18.844655 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:18.844935 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:18.844942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:18.844946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:18.845132 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:18.845525 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:18.845532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:18.845752 (+ 220us) client_negotiation.cc:769] Sending connection context
0812 11:11:18.845933 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 11:11:18.846233 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":120,"threads_started":1}
I20260812 11:11:18.848196 29007 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36269
I20260812 11:11:18.848481 29007 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:18.849038 29007 heartbeater.cc:511] Master 127.23.97.254:36269 requested a full tablet report, sending...
I20260812 11:11:18.850238 28401 ts_manager.cc:194] Registered new tserver with Master: 5b0158b6e47e411fad638cb0c37465a9 (127.23.97.194:44945)
I20260812 11:11:18.851014 28401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:40509
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:18.871064 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:18.981524 29018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:18.981794 29018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:18.981871 29018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:18.985782 29018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:18.985865 29018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:18.985955 29018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:18.990995 29018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:18.991945 29018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:18.992954 29018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:19.002473 29030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:19.002481 29033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:19.002487 29031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:19.002751 29018 server_base.cc:1034] running on GCE node
I20260812 11:11:19.003139 29018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:19.003758 29018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:19.004940 29018 hybrid_clock.cc:648] HybridClock initialized: now 1786533079004907 us; error 46 us; skew 500 ppm
Aug 12 11:11:19 dist-test-slave-zpvx krb5kdc[28320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533079, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:19.007614 29018 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:19.008569 29018 webserver.cc:460] Webserver started at http://127.23.97.195:34675/ using document root <none> and password file <none>
I20260812 11:11:19.009135 29018 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:19.009232 29018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:19.009464 29018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:19.011457 29018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "e13709a5dfcd4bdea6cde3db4eb37a12"
format_stamp: "Formatted at 2026-08-12 11:11:19 on dist-test-slave-zpvx"
server_key: "e6fe0e1b4cdea06124c2117b00962b35"
server_key_iv: "92564be73052c4f1bd92eec51bc09d0e"
server_key_version: "encryptionkey@0"
I20260812 11:11:19.012028 29018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "e13709a5dfcd4bdea6cde3db4eb37a12"
format_stamp: "Formatted at 2026-08-12 11:11:19 on dist-test-slave-zpvx"
server_key: "e6fe0e1b4cdea06124c2117b00962b35"
server_key_iv: "92564be73052c4f1bd92eec51bc09d0e"
server_key_version: "encryptionkey@0"
I20260812 11:11:19.015628 29018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260812 11:11:19.018082 29042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:19.019269 29018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:19.019412 29018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "e13709a5dfcd4bdea6cde3db4eb37a12"
format_stamp: "Formatted at 2026-08-12 11:11:19 on dist-test-slave-zpvx"
server_key: "e6fe0e1b4cdea06124c2117b00962b35"
server_key_iv: "92564be73052c4f1bd92eec51bc09d0e"
server_key_version: "encryptionkey@0"
I20260812 11:11:19.019529 29018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:19.037772 29018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:19.040347 29018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:19.040601 29018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:19.041173 29018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:19.041958 29018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:19.042032 29018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:19.042109 29018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:19.042160 29018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:19.051126 29018 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:45951
I20260812 11:11:19.051169 29245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:45951 every 8 connection(s)
I20260812 11:11:19.052002 29018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:19.056948 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29018
I20260812 11:11:19.057083 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:19.057400 23943 external_mini_cluster.cc:1428] Setting key ccd4243166f48a4b0ee83b512abc011f
Aug 12 11:11:19 dist-test-slave-zpvx krb5kdc[28320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533079, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:19.065737 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:19.053704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:58317 (local address 127.23.97.254:36269)
0812 11:11:19.053843 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:19.053854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:19.055057 (+ 1203us) server_negotiation.cc:407] Connection header received
0812 11:11:19.055960 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:19.055964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:19.056026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:19.056117 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:19.057449 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:19.058157 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:19.059565 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:19.059810 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:19.059923 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:19.062253 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:19.062259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:19.062264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:19.062298 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:19.063569 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:19.064124 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:19.064132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:19.064134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:19.064211 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:19.064617 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:19.064621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:19.064622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:19.064805 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:19.064920 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:19.065398 (+ 478us) server_negotiation.cc:299] Negotiation successful
0812 11:11:19.065564 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:11:19.066447 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:19.053989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36269 (local address 127.23.97.195:58317)
0812 11:11:19.054909 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:19.054951 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:19.055734 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:19.056304 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:19.056311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:19.056716 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:19.057289 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:19.057305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:19.058309 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:19.058314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:19.059435 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:19.059445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:19.059961 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:19.059965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:19.060148 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:19.060675 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:19.060690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:19.062051 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:19.063690 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:19.063696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:19.063710 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:19.064010 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:19.064303 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:19.064307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:19.064309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:19.064511 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:19.064921 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:19.064929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:19.065156 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 11:11:19.065338 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 11:11:19.065662 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":590,"threads_started":1}
I20260812 11:11:19.067513 29247 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36269
I20260812 11:11:19.067749 29247 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:19.068208 29247 heartbeater.cc:511] Master 127.23.97.254:36269 requested a full tablet report, sending...
I20260812 11:11:19.069217 28401 ts_manager.cc:194] Registered new tserver with Master: e13709a5dfcd4bdea6cde3db4eb37a12 (127.23.97.195:45951)
I20260812 11:11:19.069818 28401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:58317
I20260812 11:11:19.071123 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:11:19.085412 28475 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:19.084111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.23.97.254:36269)
0812 11:11:19.084231 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:19.084234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:19.084450 (+ 216us) server_negotiation.cc:407] Connection header received
0812 11:11:19.084572 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:19.084574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:19.084611 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:19.084680 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:19.085284 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55458: BlockingRecv error: recv got EOF from 127.0.0.1:55458 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:11:19.086514 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28528
I20260812 11:11:19.121397 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28775
I20260812 11:11:19.133805 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29018
I20260812 11:11:19.142611 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28347
2026-08-12T11:11:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2449 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:19 dist-test-slave-zpvx krb5kdc[29277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:19 dist-test-slave-zpvx krb5kdc[29277](info): set up 2 sockets
Aug 12 11:11:19 dist-test-slave-zpvx krb5kdc[29277](info): commencing operation
krb5kdc: starting...
W20260812 11:11:20.449532 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:20.571604 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46403
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:20.683815 29304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:20.684059 29304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:20.684106 29304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:20.690796 29304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:20.690876 29304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:20.690908 29304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:20.690945 29304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:20.690977 29304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:20.696190 29304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:20.697249 29304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:20.698225 29304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:20.706411 29317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:20.706414 29319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:20.706410 29316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:20.706678 29304 server_base.cc:1034] running on GCE node
I20260812 11:11:20.707301 29304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:20.708318 29304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:20.709568 29304 hybrid_clock.cc:648] HybridClock initialized: now 1786533080709551 us; error 34 us; skew 500 ppm
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533080, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:20.716626 29304 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:20.717533 29304 webserver.cc:460] Webserver started at http://127.23.97.254:38409/ using document root <none> and password file <none>
I20260812 11:11:20.718078 29304 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:20.718138 29304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:20.718354 29304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:20.720432 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "e929416037cb4c28a398b1638a4ad5ae"
format_stamp: "Formatted at 2026-08-12 11:11:20 on dist-test-slave-zpvx"
server_key: "8b3e66e76843d726a74022d8e6d4607f"
server_key_iv: "28cbfa26f9911dba0fc4ebfb67ff98e6"
server_key_version: "encryptionkey@0"
I20260812 11:11:20.721017 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "e929416037cb4c28a398b1638a4ad5ae"
format_stamp: "Formatted at 2026-08-12 11:11:20 on dist-test-slave-zpvx"
server_key: "8b3e66e76843d726a74022d8e6d4607f"
server_key_iv: "28cbfa26f9911dba0fc4ebfb67ff98e6"
server_key_version: "encryptionkey@0"
I20260812 11:11:20.724637 29304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:20.726941 29330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:20.728156 29304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:20.728295 29304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e929416037cb4c28a398b1638a4ad5ae"
format_stamp: "Formatted at 2026-08-12 11:11:20 on dist-test-slave-zpvx"
server_key: "8b3e66e76843d726a74022d8e6d4607f"
server_key_iv: "28cbfa26f9911dba0fc4ebfb67ff98e6"
server_key_version: "encryptionkey@0"
I20260812 11:11:20.728441 29304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:20.737365 29304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:20.740487 29304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:20.740698 29304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:20.748565 29304 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:46403
I20260812 11:11:20.748605 29411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:46403 every 8 connection(s)
I20260812 11:11:20.749559 29304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:20.752318 29412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:20.757637 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29304
I20260812 11:11:20.757812 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:20.757932 29412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: Bootstrap starting.
I20260812 11:11:20.758127 23943 external_mini_cluster.cc:1428] Setting key a1144ccd4269fd0c8d6a08f2ccfe4a55
I20260812 11:11:20.760828 29412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:20.761741 29412 log.cc:824] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:20.763988 29412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: No bootstrap required, opened a new log
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:20.768512 29412 raft_consensus.cc:348] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } }
I20260812 11:11:20.768769 29412 raft_consensus.cc:374] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:20.768858 29412 raft_consensus.cc:729] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e929416037cb4c28a398b1638a4ad5ae, State: Initialized, Role: FOLLOWER
I20260812 11:11:20.769380 29412 consensus_queue.cc:260] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } }
I20260812 11:11:20.769513 29412 raft_consensus.cc:388] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:20.769618 29412 raft_consensus.cc:482] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:20.769728 29412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:20.770542 29412 raft_consensus.cc:504] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } }
I20260812 11:11:20.770931 29412 leader_election.cc:302] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e929416037cb4c28a398b1638a4ad5ae; no voters:
I20260812 11:11:20.771328 29412 leader_election.cc:288] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:20.771397 29423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:20.771610 29423 raft_consensus.cc:686] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [term 1 LEADER]: Becoming Leader. State: Replica: e929416037cb4c28a398b1638a4ad5ae, State: Running, Role: LEADER
I20260812 11:11:20.771925 29423 consensus_queue.cc:237] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } }
I20260812 11:11:20.772671 29412 sys_catalog.cc:564] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:20.774288 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:20.759438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.23.97.254:46403)
0812 11:11:20.759876 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:20.759895 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:20.759937 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:11:20.760828 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:20.760854 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:20.761301 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:20.761686 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:20.762579 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:20.764073 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:20.765002 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:20.765712 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:20.765891 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:20.768613 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:20.768618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:20.768631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:20.768658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:20.770576 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:20.771143 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:20.771149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:20.771154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:20.771268 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:20.771597 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:20.771601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:20.771603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:20.772123 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:20.772342 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:20.773029 (+ 687us) server_negotiation.cc:299] Negotiation successful
0812 11:11:20.773329 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260812 11:11:20.775318 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e929416037cb4c28a398b1638a4ad5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } } }
I20260812 11:11:20.775730 29427 sys_catalog.cc:455] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader e929416037cb4c28a398b1638a4ad5ae. Latest consensus state: current_term: 1 leader_uuid: "e929416037cb4c28a398b1638a4ad5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e929416037cb4c28a398b1638a4ad5ae" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46403 } } }
I20260812 11:11:20.775826 29427 sys_catalog.cc:458] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:20.775434 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:20.777357 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 11:11:20.778111 29445 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:11:20.778191 29445 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:11:20.778252 29446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:20.779146 29446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:20.785027 29446 catalog_manager.cc:1347] Generated new cluster ID: 2bdd1c9d95134b5e928c666af93a182c
I20260812 11:11:20.785099 29446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:20.799505 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:20.794036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.23.97.254:46403)
0812 11:11:20.794127 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:20.794130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:20.794393 (+ 263us) server_negotiation.cc:407] Connection header received
0812 11:11:20.794483 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:20.794486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:20.794534 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:20.794624 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:20.795176 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:20.795961 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:20.797152 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:20.797540 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:20.797658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:20.798539 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:20.798544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:20.798547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:20.799064 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:20.799098 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:20.799118 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:20.799209 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:20.799314 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 11:11:20.799371 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 11:11:20.805914 29446 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:20.806792 29446 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:20.813647 29446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e929416037cb4c28a398b1638a4ad5ae: Generated new TSK 0
I20260812 11:11:20.814335 29446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:11:20.825393 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29304
2026-08-12T11:11:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1686 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29465](info): set up 2 sockets
Aug 12 11:11:20 dist-test-slave-zpvx krb5kdc[29465](info): commencing operation
krb5kdc: starting...
W20260812 11:11:22.137770 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:22.262907 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34955
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:22.373411 29494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:22.373713 29494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:22.373782 29494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:22.377784 29494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:22.377864 29494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:22.377902 29494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:22.377961 29494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:22.378001 29494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:22.383028 29494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:22.384007 29494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:22.385205 29494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:22.393815 29512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:22.393791 29505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:22.393814 29508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:22.394006 29494 server_base.cc:1034] running on GCE node
I20260812 11:11:22.394697 29494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:22.395702 29494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:22.396865 29494 hybrid_clock.cc:648] HybridClock initialized: now 1786533082396846 us; error 29 us; skew 500 ppm
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:22.399633 29494 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:22.400588 29494 webserver.cc:460] Webserver started at http://127.23.97.254:40471/ using document root <none> and password file <none>
I20260812 11:11:22.401134 29494 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:22.401269 29494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:22.401503 29494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:22.403512 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "72d5d163ca1c4258b24adb11d3c1f892"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "5da61766d0f91556b1a13e2efe213a95"
server_key_iv: "675d99d34ca17f4ec2d693467122727b"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.404074 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "72d5d163ca1c4258b24adb11d3c1f892"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "5da61766d0f91556b1a13e2efe213a95"
server_key_iv: "675d99d34ca17f4ec2d693467122727b"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.407692 29494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:11:22.410144 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.411268 29494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:22.411512 29494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "72d5d163ca1c4258b24adb11d3c1f892"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "5da61766d0f91556b1a13e2efe213a95"
server_key_iv: "675d99d34ca17f4ec2d693467122727b"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.411624 29494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:22.456059 29494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:22.459559 29494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:22.459805 29494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:22.467334 29494 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34955
I20260812 11:11:22.467674 29626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34955 every 8 connection(s)
I20260812 11:11:22.468314 29494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:22.469135 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29494
I20260812 11:11:22.469275 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:22.469549 23943 external_mini_cluster.cc:1428] Setting key 778c3d4cfad33f7c9b8b1404d40b10bf
I20260812 11:11:22.471496 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:22.478216 29627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: Bootstrap starting.
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:22.480909 29627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:22.481817 29627 log.cc:824] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:22.484097 29627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: No bootstrap required, opened a new log
I20260812 11:11:22.484462 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:22.470969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.23.97.254:34955)
0812 11:11:22.471384 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:22.471399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:22.471435 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:11:22.472163 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:22.472180 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:22.472564 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:22.472855 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:22.473807 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.474804 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.475969 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.476499 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.476615 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:22.479401 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:22.479411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:22.479427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:22.479462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:22.481405 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.481844 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.481850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.481856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.481991 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.482337 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.482340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.482341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.482795 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:22.482975 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:22.483258 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 11:11:22.483532 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":141,"threads_started":1}
I20260812 11:11:22.487866 29627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } }
I20260812 11:11:22.488065 29627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:22.488098 29627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d5d163ca1c4258b24adb11d3c1f892, State: Initialized, Role: FOLLOWER
I20260812 11:11:22.488502 29627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } }
I20260812 11:11:22.488619 29627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:22.488662 29627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:22.488736 29627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:22.489539 29627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } }
I20260812 11:11:22.489822 29627 leader_election.cc:302] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d5d163ca1c4258b24adb11d3c1f892; no voters:
I20260812 11:11:22.490113 29627 leader_election.cc:288] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:22.490218 29634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:22.490482 29634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [term 1 LEADER]: Becoming Leader. State: Replica: 72d5d163ca1c4258b24adb11d3c1f892, State: Running, Role: LEADER
I20260812 11:11:22.490784 29634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } }
I20260812 11:11:22.491273 29627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:22.492445 29635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d5d163ca1c4258b24adb11d3c1f892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } } }
I20260812 11:11:22.492574 29635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:22.492874 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d5d163ca1c4258b24adb11d3c1f892. Latest consensus state: current_term: 1 leader_uuid: "72d5d163ca1c4258b24adb11d3c1f892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d5d163ca1c4258b24adb11d3c1f892" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34955 } } }
I20260812 11:11:22.492995 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892 [sys.catalog]: This master's current role is: LEADER
W20260812 11:11:22.495873 29660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:11:22.495939 29660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:11:22.496027 29648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:22.496918 29648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:22.501885 29648 catalog_manager.cc:1347] Generated new cluster ID: 66fcfc6063184fd6bd2e8e0bdb7ac3c1
I20260812 11:11:22.501947 29648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:22.511405 29648 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:22.512306 29648 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:22.518859 29648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d5d163ca1c4258b24adb11d3c1f892: Generated new TSK 0
I20260812 11:11:22.519460 29648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:22.550781 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:22.662933 29667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:22.663172 29667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:22.663219 29667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:22.667599 29667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:22.667654 29667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:22.667737 29667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:22.672892 29667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:22.673858 29667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:22.674860 29667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:22.684571 29681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:22.684742 29667 server_base.cc:1034] running on GCE node
W20260812 11:11:22.684571 29684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:22.686175 29682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:22.686715 29667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:22.687588 29667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:22.688797 29667 hybrid_clock.cc:648] HybridClock initialized: now 1786533082688764 us; error 40 us; skew 500 ppm
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:22.692471 29667 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:22.693461 29667 webserver.cc:460] Webserver started at http://127.23.97.193:35629/ using document root <none> and password file <none>
I20260812 11:11:22.694067 29667 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:22.694129 29667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:22.694286 29667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:22.696141 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "009e5b50bb004e8ca27599403528641c"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "64d72994f49878588b0a9cf349241021"
server_key_iv: "34f7514c9566f1d0f8506dd7bcd5969f"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.696699 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "009e5b50bb004e8ca27599403528641c"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "64d72994f49878588b0a9cf349241021"
server_key_iv: "34f7514c9566f1d0f8506dd7bcd5969f"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.700057 29667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:11:22.702350 29694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.703504 29667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:22.703595 29667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "009e5b50bb004e8ca27599403528641c"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "64d72994f49878588b0a9cf349241021"
server_key_iv: "34f7514c9566f1d0f8506dd7bcd5969f"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.703675 29667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:22.735088 29667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:22.737951 29667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:22.738104 29667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:22.738633 29667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:22.739403 29667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:22.739459 29667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.739497 29667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:22.739519 29667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.748680 29667 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:42927
I20260812 11:11:22.748744 29878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:42927 every 8 connection(s)
I20260812 11:11:22.749614 29667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:22.757210 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29667
I20260812 11:11:22.757356 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:22.757627 23943 external_mini_cluster.cc:1428] Setting key 4efd03bedeb25272a120b6d9630e3a0b
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:22.764801 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:22.751415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54745 (local address 127.23.97.254:34955)
0812 11:11:22.751551 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:22.751558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:22.752407 (+ 849us) server_negotiation.cc:407] Connection header received
0812 11:11:22.754188 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:22.754206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:22.754275 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:22.754378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:22.756016 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.756723 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.757982 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.758311 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.758422 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:22.761030 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:22.761034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:22.761036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:22.761061 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:22.762420 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.763054 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.763057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.763059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.763166 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.763561 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.763564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.763565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.763729 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:22.763805 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:22.764495 (+ 690us) server_negotiation.cc:299] Negotiation successful
0812 11:11:22.764669 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:22.765923 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:22.751774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34955 (local address 127.23.97.193:54745)
0812 11:11:22.752203 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:22.752265 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:22.753929 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:22.754619 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:22.754628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:22.755178 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:22.755837 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:22.755860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.756844 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.756847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:22.757859 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:22.757869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.758475 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.758480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:22.758667 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:22.759368 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:22.759384 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:22.760865 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:22.762552 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:22.762561 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:22.762579 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:22.762952 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:22.763267 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:22.763273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:22.763278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:22.763466 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:22.763835 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:22.763844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:22.764190 (+ 346us) client_negotiation.cc:769] Sending connection context
0812 11:11:22.764431 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 11:11:22.764806 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260812 11:11:22.767172 29880 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34955
I20260812 11:11:22.767400 29880 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:22.767877 29880 heartbeater.cc:511] Master 127.23.97.254:34955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:22.769428 29548 ts_manager.cc:194] Registered new tserver with Master: 009e5b50bb004e8ca27599403528641c (127.23.97.193:42927)
I20260812 11:11:22.770537 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:54745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:22.787146 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:22.898310 29889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:22.898563 29889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:22.898645 29889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:22.902774 29889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:22.902844 29889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:22.902923 29889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:22.908186 29889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:22.909195 29889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:22.910261 29889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:22.919529 29901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:22.919515 29899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:22.919518 29903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:22.919724 29889 server_base.cc:1034] running on GCE node
I20260812 11:11:22.920197 29889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:22.920830 29889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:22.921998 29889 hybrid_clock.cc:648] HybridClock initialized: now 1786533082921978 us; error 28 us; skew 500 ppm
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:22.925156 29889 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:22.926017 29889 webserver.cc:460] Webserver started at http://127.23.97.194:45679/ using document root <none> and password file <none>
I20260812 11:11:22.926553 29889 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:22.926643 29889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:22.926867 29889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:22.928848 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "1ad424d4d40a4ba7aab87c44c9b5aaac"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "bbcdedc7b9b3b25258f02947be700271"
server_key_iv: "1ac953459d325f672fb65803d221f3ec"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.929423 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "1ad424d4d40a4ba7aab87c44c9b5aaac"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "bbcdedc7b9b3b25258f02947be700271"
server_key_iv: "1ac953459d325f672fb65803d221f3ec"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.932976 29889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:22.935205 29911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.936417 29889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:22.936555 29889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "1ad424d4d40a4ba7aab87c44c9b5aaac"
format_stamp: "Formatted at 2026-08-12 11:11:22 on dist-test-slave-zpvx"
server_key: "bbcdedc7b9b3b25258f02947be700271"
server_key_iv: "1ac953459d325f672fb65803d221f3ec"
server_key_version: "encryptionkey@0"
I20260812 11:11:22.936662 29889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:22.954051 29889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:22.957342 29889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:22.957559 29889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:22.958209 29889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:22.959060 29889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:22.959131 29889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.959196 29889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:22.959256 29889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:22.968183 29889 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:37357
I20260812 11:11:22.968228 30119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:37357 every 8 connection(s)
I20260812 11:11:22.969026 29889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:22.973443 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29889
I20260812 11:11:22.973578 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:22.973884 23943 external_mini_cluster.cc:1428] Setting key 91e7c7ed9399987872da036d945a285b
Aug 12 11:11:22 dist-test-slave-zpvx krb5kdc[29465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:22.983319 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:22.970718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:47751 (local address 127.23.97.254:34955)
0812 11:11:22.970852 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:22.970855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:22.971597 (+ 742us) server_negotiation.cc:407] Connection header received
0812 11:11:22.972897 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:22.972900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:22.972951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:22.973027 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:22.974491 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.975231 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.976931 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.977158 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.977245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:22.979732 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:22.979736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:22.979738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:22.979770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:22.981275 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.981770 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.981774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.981776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.981858 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:22.982197 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:22.982200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:22.982202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:22.982380 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:22.982494 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:22.983031 (+ 537us) server_negotiation.cc:299] Negotiation successful
0812 11:11:22.983150 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:11:22.984045 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:22.971019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34955 (local address 127.23.97.194:47751)
0812 11:11:22.971443 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:22.971483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:22.972694 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:22.973169 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:22.973198 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:22.973702 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:22.974332 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:22.974349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.975390 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.975395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:22.976800 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:22.976814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:22.977317 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:22.977323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:22.977548 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:22.978171 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:22.978187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:22.979562 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:22.981393 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:22.981398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:22.981412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:22.981664 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:22.981953 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:22.981957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:22.981959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:22.982095 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:22.982495 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:22.982502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:22.982763 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 11:11:22.982985 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 11:11:22.983288 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:11:22.985225 30120 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34955
I20260812 11:11:22.985448 30120 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:22.985918 30120 heartbeater.cc:511] Master 127.23.97.254:34955 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:11:22.986977 29548 ts_manager.cc:194] Registered new tserver with Master: 1ad424d4d40a4ba7aab87c44c9b5aaac (127.23.97.194:37357)
I20260812 11:11:22.987586 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:47751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:23.003557 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:23.115109 30128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:23.115442 30128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:23.115530 30128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:23.119668 30128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:23.119763 30128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:23.119848 30128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:23.125002 30128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:23.125973 30128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:23.127007 30128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:23.136463 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:23.136510 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:23.136464 30148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:23.136627 30128 server_base.cc:1034] running on GCE node
I20260812 11:11:23.137106 30128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:23.137702 30128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:23.138906 30128 hybrid_clock.cc:648] HybridClock initialized: now 1786533083138887 us; error 28 us; skew 500 ppm
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:23.141774 30128 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:23.142549 30128 webserver.cc:460] Webserver started at http://127.23.97.195:36385/ using document root <none> and password file <none>
I20260812 11:11:23.143056 30128 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:23.143115 30128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:23.143277 30128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:23.145289 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "79a22ebbbad446a68374761ad59a7a04"
format_stamp: "Formatted at 2026-08-12 11:11:23 on dist-test-slave-zpvx"
server_key: "70c05726d10b188e4570c05a05b5447c"
server_key_iv: "8b69dc57ebdcac4b531346c5402e260a"
server_key_version: "encryptionkey@0"
I20260812 11:11:23.145805 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "79a22ebbbad446a68374761ad59a7a04"
format_stamp: "Formatted at 2026-08-12 11:11:23 on dist-test-slave-zpvx"
server_key: "70c05726d10b188e4570c05a05b5447c"
server_key_iv: "8b69dc57ebdcac4b531346c5402e260a"
server_key_version: "encryptionkey@0"
I20260812 11:11:23.149442 30128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:11:23.151688 30160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:23.152910 30128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:23.153003 30128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "79a22ebbbad446a68374761ad59a7a04"
format_stamp: "Formatted at 2026-08-12 11:11:23 on dist-test-slave-zpvx"
server_key: "70c05726d10b188e4570c05a05b5447c"
server_key_iv: "8b69dc57ebdcac4b531346c5402e260a"
server_key_version: "encryptionkey@0"
I20260812 11:11:23.153082 30128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:23.188028 30128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:23.190744 30128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:23.190888 30128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:23.191506 30128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:23.192292 30128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:23.192348 30128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:23.192468 30128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:23.192497 30128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:23.201028 30128 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:43023
I20260812 11:11:23.201059 30348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:43023 every 8 connection(s)
I20260812 11:11:23.201891 30128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:23.209970 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30128
I20260812 11:11:23.210108 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:23.210422 23943 external_mini_cluster.cc:1428] Setting key 5aea7d0cfb2132a46f5aea702f9f6e56
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[29465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:23.215409 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:23.203604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:37785 (local address 127.23.97.254:34955)
0812 11:11:23.203742 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:23.203746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:23.204578 (+ 832us) server_negotiation.cc:407] Connection header received
0812 11:11:23.205933 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:23.205937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:23.205994 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:23.206102 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:23.207289 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.207817 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.209025 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.209254 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.209367 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:23.211990 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:23.211994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:23.211997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:23.212023 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:23.213589 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:23.214050 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:23.214053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:23.214054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:23.214128 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:23.214424 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:23.214429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:23.214431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:23.214628 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:23.214729 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:23.215202 (+ 473us) server_negotiation.cc:299] Negotiation successful
0812 11:11:23.215298 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:11:23.216194 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:23.203951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34955 (local address 127.23.97.195:37785)
0812 11:11:23.204429 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:23.204484 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:23.205729 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:23.206279 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:23.206286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:23.206618 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:23.207141 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:23.207155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.207947 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.207953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:23.208890 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:23.208900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.209403 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.209407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:23.209566 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:23.210176 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:23.210202 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:23.211826 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:23.213690 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:23.213695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:23.213706 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:23.213961 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:23.214204 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:23.214206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:23.214208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:23.214334 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:23.214731 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:23.214739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:23.214965 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 11:11:23.215130 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 11:11:23.215362 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260812 11:11:23.217306 30349 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34955
I20260812 11:11:23.217525 30349 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:23.217959 30349 heartbeater.cc:511] Master 127.23.97.254:34955 requested a full tablet report, sending...
I20260812 11:11:23.218924 29548 ts_manager.cc:194] Registered new tserver with Master: 79a22ebbbad446a68374761ad59a7a04 (127.23.97.195:43023)
I20260812 11:11:23.219429 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:37785
I20260812 11:11:23.224243 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[29465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533083, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[29465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533083, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:23.250451 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:23.242068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.23.97.254:34955)
0812 11:11:23.242241 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:23.242247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:23.242435 (+ 188us) server_negotiation.cc:407] Connection header received
0812 11:11:23.242606 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:23.242615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:23.242658 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:23.242746 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:23.243453 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.243963 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.244877 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:23.245095 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:23.245203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:23.247575 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:23.247578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:23.247580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:23.247602 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:23.249029 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:23.249422 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:23.249425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:23.249426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:23.249498 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:23.249734 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:23.249736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:23.249737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:23.249932 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:23.250032 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:23.250247 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 11:11:23.250335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 11:11:23.250897 29548 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56004
I20260812 11:11:23.251850 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29667
I20260812 11:11:23.266040 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29889
I20260812 11:11:23.300863 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30128
I20260812 11:11:23.314891 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29494
2026-08-12T11:11:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2484 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[30385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[30385](info): set up 2 sockets
Aug 12 11:11:23 dist-test-slave-zpvx krb5kdc[30385](info): commencing operation
krb5kdc: starting...
W20260812 11:11:24.628858 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:24 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:24.753479 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:41797
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:41797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:24.865751 30411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:24.866005 30411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:24.866044 30411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:24.870209 30411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:24.870267 30411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:24.870287 30411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:24.870304 30411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:24.870323 30411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:24.875703 30411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:41797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:41797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:24.876762 30411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:24.877777 30411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:24.886204 30422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:24.886288 30424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:24.886252 30421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:24.886426 30411 server_base.cc:1034] running on GCE node
I20260812 11:11:24.887133 30411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:24.888250 30411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:24.889441 30411 hybrid_clock.cc:648] HybridClock initialized: now 1786533084889418 us; error 38 us; skew 500 ppm
Aug 12 11:11:24 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533084, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:24.892195 30411 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:24.893110 30411 webserver.cc:460] Webserver started at http://127.23.97.254:39551/ using document root <none> and password file <none>
I20260812 11:11:24.893658 30411 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:24.893774 30411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:24.893998 30411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:24.895987 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e"
format_stamp: "Formatted at 2026-08-12 11:11:24 on dist-test-slave-zpvx"
server_key: "4395314aff99ccc58aecaa5875997237"
server_key_iv: "c15f0c1c9c84b1f824f7fbd392f9709a"
server_key_version: "encryptionkey@0"
I20260812 11:11:24.896605 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e"
format_stamp: "Formatted at 2026-08-12 11:11:24 on dist-test-slave-zpvx"
server_key: "4395314aff99ccc58aecaa5875997237"
server_key_iv: "c15f0c1c9c84b1f824f7fbd392f9709a"
server_key_version: "encryptionkey@0"
I20260812 11:11:24.900163 30411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:11:24.902554 30433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:24.903697 30411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:24.903846 30411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e"
format_stamp: "Formatted at 2026-08-12 11:11:24 on dist-test-slave-zpvx"
server_key: "4395314aff99ccc58aecaa5875997237"
server_key_iv: "c15f0c1c9c84b1f824f7fbd392f9709a"
server_key_version: "encryptionkey@0"
I20260812 11:11:24.903952 30411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:24.913542 30411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:24.916762 30411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:24.916961 30411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:24.924299 30411 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:41797
I20260812 11:11:24.924431 30524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:41797 every 8 connection(s)
I20260812 11:11:24.925264 30411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:24.928113 30526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:24.929157 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30411
I20260812 11:11:24.929291 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:24.929570 23943 external_mini_cluster.cc:1428] Setting key 69bf1b60d5b3e6efa0c680725fb3581d
I20260812 11:11:24.934427 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e: Bootstrap starting.
I20260812 11:11:24.936616 30526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:24.937335 30526 log.cc:824] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:11:24 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:24.939221 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e: No bootstrap required, opened a new log
I20260812 11:11:24.942649 30526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } }
I20260812 11:11:24.942898 30526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:24.942992 30526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d408b0cb5ea4bf4a66dd3b115c3165e, State: Initialized, Role: FOLLOWER
I20260812 11:11:24.943444 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:24.930937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.23.97.254:41797)
0812 11:11:24.931385 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:24.931399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:24.931433 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:11:24.932058 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:24.932074 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:24.932442 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:24.932779 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:24.933578 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:24.934596 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:24.935440 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:24.935911 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:24.936090 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:24.938722 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:24.938729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:24.938747 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:24.938775 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:24.940590 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:24.941039 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:24.941044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:24.941050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:24.941167 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:24.941463 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:24.941467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:24.941469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:24.941836 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:24.942049 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:24.942313 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 11:11:24.942577 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260812 11:11:24.943454 30526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } }
I20260812 11:11:24.943704 30526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:24.943780 30526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:24.943877 30526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:24.944909 30526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } }
I20260812 11:11:24.945242 30526 leader_election.cc:302] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d408b0cb5ea4bf4a66dd3b115c3165e; no voters:
I20260812 11:11:24.945578 30526 leader_election.cc:288] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:24.945645 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:24.945830 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [term 1 LEADER]: Becoming Leader. State: Replica: 4d408b0cb5ea4bf4a66dd3b115c3165e, State: Running, Role: LEADER
I20260812 11:11:24.946102 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } }
I20260812 11:11:24.946799 30526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:24.947822 30534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d408b0cb5ea4bf4a66dd3b115c3165e. Latest consensus state: current_term: 1 leader_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } } }
I20260812 11:11:24.947973 30534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:24.948262 30536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d408b0cb5ea4bf4a66dd3b115c3165e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 41797 } } }
I20260812 11:11:24.948354 30536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:24.948534 30546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:24.951560 30546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:24.957036 30546 catalog_manager.cc:1347] Generated new cluster ID: 49bb4d5fd456409a96e3868be6a7a509
I20260812 11:11:24.957105 30546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:24.999521 30546 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:25.000609 30546 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:25.005723 30546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d408b0cb5ea4bf4a66dd3b115c3165e: Generated new TSK 0
I20260812 11:11:25.006462 30546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:25.044114 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:25.162348 30563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:25.162592 30563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:25.162642 30563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:25.166676 30563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:25.166733 30563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:25.166823 30563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:25.172210 30563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:25.173245 30563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:25.174266 30563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:25.183773 30577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.183784 30580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.183785 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:25.184141 30563 server_base.cc:1034] running on GCE node
I20260812 11:11:25.184573 30563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:25.185189 30563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:25.186372 30563 hybrid_clock.cc:648] HybridClock initialized: now 1786533085186348 us; error 46 us; skew 500 ppm
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:25.189204 30563 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:25.190063 30563 webserver.cc:460] Webserver started at http://127.23.97.193:41931/ using document root <none> and password file <none>
I20260812 11:11:25.190619 30563 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:25.190714 30563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:25.190940 30563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:25.192991 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "05e5e29f5c614ff8866abf1780bd645d"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "508ad2ae113c9d36f593343bc00c757f"
server_key_iv: "a2d7922dae87400bc6c7365f214d31a3"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.193564 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "05e5e29f5c614ff8866abf1780bd645d"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "508ad2ae113c9d36f593343bc00c757f"
server_key_iv: "a2d7922dae87400bc6c7365f214d31a3"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.197214 30563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:25.199522 30591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.200727 30563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:25.200860 30563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "05e5e29f5c614ff8866abf1780bd645d"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "508ad2ae113c9d36f593343bc00c757f"
server_key_iv: "a2d7922dae87400bc6c7365f214d31a3"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.200968 30563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:25.212025 30563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:25.214670 30563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:25.214865 30563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:25.215423 30563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:25.216199 30563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:25.216271 30563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.216339 30563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:25.216451 30563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.225566 30563 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:39063
I20260812 11:11:25.225601 30774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:39063 every 8 connection(s)
I20260812 11:11:25.226387 30563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:25.233016 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30563
I20260812 11:11:25.233151 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:25.233490 23943 external_mini_cluster.cc:1428] Setting key 7aa0f8843b16b71cdfb91e11ea265f55
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:25.240407 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.228039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54697 (local address 127.23.97.254:41797)
0812 11:11:25.228178 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:25.228182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:25.229000 (+ 818us) server_negotiation.cc:407] Connection header received
0812 11:11:25.230307 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:25.230309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:25.230356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:25.230430 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:25.231509 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.232037 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.233434 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.233703 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.233789 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.236349 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:25.236353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:25.236357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:25.236428 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:25.238173 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.238709 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.238716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.238720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.238817 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.239133 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.239139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.239143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.239401 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:25.239508 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:25.239955 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 11:11:25.240117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:11:25.241058 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.228421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:41797 (local address 127.23.97.193:54697)
0812 11:11:25.228850 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:25.228889 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:25.230106 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:25.230587 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:25.230593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:25.230939 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:25.231369 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.231382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.232146 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.232148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.233299 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.233311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.233862 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.233867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.234020 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.234591 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:25.234606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:25.236175 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:25.238319 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.238324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.238334 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.238600 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.238910 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.238912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.238913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.239038 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.239505 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:25.239510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:25.239713 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 11:11:25.239867 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 11:11:25.240174 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260812 11:11:25.242305 30775 heartbeater.cc:347] Connected to a master server at 127.23.97.254:41797
I20260812 11:11:25.242558 30775 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:25.243108 30775 heartbeater.cc:511] Master 127.23.97.254:41797 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:25.244764 30461 ts_manager.cc:194] Registered new tserver with Master: 05e5e29f5c614ff8866abf1780bd645d (127.23.97.193:39063)
I20260812 11:11:25.245982 30461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:54697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:25.263254 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:25.372363 30792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:25.372612 30792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:25.372660 30792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:25.376663 30792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:25.376714 30792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:25.376799 30792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:25.383104 30792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:25.384055 30792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:25.385048 30792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:25.394522 30802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.394565 30801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.394550 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:25.394675 30792 server_base.cc:1034] running on GCE node
I20260812 11:11:25.395141 30792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:25.395684 30792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:25.396847 30792 hybrid_clock.cc:648] HybridClock initialized: now 1786533085396826 us; error 31 us; skew 500 ppm
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:25.399545 30792 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:25.400482 30792 webserver.cc:460] Webserver started at http://127.23.97.194:32861/ using document root <none> and password file <none>
I20260812 11:11:25.401027 30792 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:25.401120 30792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:25.401381 30792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:25.403338 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "e1aaf3e521ed471585bcabc8c2cd2520"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "5af64555bbe1bcf334560728b6fbfbee"
server_key_iv: "10765b2cf2ccfaa9b2f6986a946e2377"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.403893 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "e1aaf3e521ed471585bcabc8c2cd2520"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "5af64555bbe1bcf334560728b6fbfbee"
server_key_iv: "10765b2cf2ccfaa9b2f6986a946e2377"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.407544 30792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20260812 11:11:25.410096 30816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.411322 30792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:25.411458 30792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "e1aaf3e521ed471585bcabc8c2cd2520"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "5af64555bbe1bcf334560728b6fbfbee"
server_key_iv: "10765b2cf2ccfaa9b2f6986a946e2377"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.411574 30792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:25.425232 30792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:25.427966 30792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:25.428181 30792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:25.428792 30792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:25.429941 30792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:25.430002 30792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.430061 30792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:25.430094 30792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.438603 30792 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:45909
I20260812 11:11:25.438637 31013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:45909 every 8 connection(s)
I20260812 11:11:25.439523 30792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:25.449419 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30792
I20260812 11:11:25.449545 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:25.449904 23943 external_mini_cluster.cc:1428] Setting key 70dc6f7f91cb96d91e7c2d029cd1d1c4
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:25.454854 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.441387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:57675 (local address 127.23.97.254:41797)
0812 11:11:25.441516 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:25.441521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:25.442332 (+ 811us) server_negotiation.cc:407] Connection header received
0812 11:11:25.443736 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:25.443740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:25.443799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:25.443906 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:25.445174 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.445715 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.446900 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.447148 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.447261 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.450778 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:25.450782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:25.450786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:25.450811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:25.452616 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.453122 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.453129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.453133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.453229 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.453604 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.453610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.453615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.453879 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:25.453975 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:25.454629 (+ 654us) server_negotiation.cc:299] Negotiation successful
0812 11:11:25.454727 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:11:25.455832 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.441742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:41797 (local address 127.23.97.194:57675)
0812 11:11:25.442193 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:25.442234 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:25.443522 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:25.444110 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:25.444118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:25.444538 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:25.445003 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.445017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.445849 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.445857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.446743 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.446753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.447315 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.447319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.447519 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.448251 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:25.448267 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:25.450628 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:25.452755 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.452760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.452772 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.453022 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.453357 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.453360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.453361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.453501 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.454014 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:25.454021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:25.454363 (+ 342us) client_negotiation.cc:769] Sending connection context
0812 11:11:25.454602 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 11:11:25.454962 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260812 11:11:25.457026 31016 heartbeater.cc:347] Connected to a master server at 127.23.97.254:41797
I20260812 11:11:25.457283 31016 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:25.457751 31016 heartbeater.cc:511] Master 127.23.97.254:41797 requested a full tablet report, sending...
I20260812 11:11:25.458823 30461 ts_manager.cc:194] Registered new tserver with Master: e1aaf3e521ed471585bcabc8c2cd2520 (127.23.97.194:45909)
I20260812 11:11:25.459640 30461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:57675
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:25.480067 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:25.593330 31025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:25.593562 31025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:25.593602 31025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:25.597536 31025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:25.597589 31025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:25.597644 31025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:25.602596 31025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:25.603467 31025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:25.604490 31025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:11:25.614080 31025 server_base.cc:1034] running on GCE node
W20260812 11:11:25.614049 31040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.614059 31039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:25.614059 31043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:25.614775 31025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:25.615386 31025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:25.616557 31025 hybrid_clock.cc:648] HybridClock initialized: now 1786533085616542 us; error 35 us; skew 500 ppm
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:25.619226 31025 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:25.620111 31025 webserver.cc:460] Webserver started at http://127.23.97.195:33229/ using document root <none> and password file <none>
I20260812 11:11:25.620728 31025 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:25.620823 31025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:25.621049 31025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:25.623014 31025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "97ee94b214654ccea536c84464caa9e8"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "bd780d8ccf58b959feb19369c98b1459"
server_key_iv: "f0f1391a76fbde988168cbc3fb3d14bb"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.623594 31025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "97ee94b214654ccea536c84464caa9e8"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "bd780d8ccf58b959feb19369c98b1459"
server_key_iv: "f0f1391a76fbde988168cbc3fb3d14bb"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.627244 31025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:11:25.629815 31065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.630982 31025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:25.631131 31025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "97ee94b214654ccea536c84464caa9e8"
format_stamp: "Formatted at 2026-08-12 11:11:25 on dist-test-slave-zpvx"
server_key: "bd780d8ccf58b959feb19369c98b1459"
server_key_iv: "f0f1391a76fbde988168cbc3fb3d14bb"
server_key_version: "encryptionkey@0"
I20260812 11:11:25.631244 31025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:25.650734 31025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:25.653637 31025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:25.653862 31025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:25.654474 31025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:25.655227 31025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:25.655275 31025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.655313 31025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:25.655335 31025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:25.663816 31025 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:41225
I20260812 11:11:25.663894 31267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:41225 every 8 connection(s)
I20260812 11:11:25.664757 31025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:25.667323 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31025
I20260812 11:11:25.667464 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:25.667820 23943 external_mini_cluster.cc:1428] Setting key 975227a6e5729373d49bb943e3a13e73
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:25.678136 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.666519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:52505 (local address 127.23.97.254:41797)
0812 11:11:25.666662 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:25.666666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:25.667591 (+ 925us) server_negotiation.cc:407] Connection header received
0812 11:11:25.669351 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:25.669354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:25.669410 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:25.669515 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:25.670790 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.671322 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.672486 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.672718 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.672833 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.675146 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:25.675150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:25.675153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:25.675180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:25.676428 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.676888 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.676890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.676892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.676966 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.677247 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.677249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.677250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.677407 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:25.677481 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:25.677924 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 11:11:25.678017 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:25.678869 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.666870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:41797 (local address 127.23.97.195:52505)
0812 11:11:25.667432 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:25.667492 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:25.669149 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:25.669691 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:25.669698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:25.670199 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:25.670651 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.670665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.671449 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.671453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.672334 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:25.672343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.672872 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.672876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:25.673049 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.673592 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:25.673609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:25.675000 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:25.676530 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.676534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.676544 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.676805 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.677044 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:25.677046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:25.677047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:25.677171 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:25.677492 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:25.677496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:25.677700 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 11:11:25.677860 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 11:11:25.678081 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":122,"threads_started":1}
I20260812 11:11:25.679891 31269 heartbeater.cc:347] Connected to a master server at 127.23.97.254:41797
I20260812 11:11:25.680119 31269 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:25.680795 31269 heartbeater.cc:511] Master 127.23.97.254:41797 requested a full tablet report, sending...
I20260812 11:11:25.681887 30461 ts_manager.cc:194] Registered new tserver with Master: 97ee94b214654ccea536c84464caa9e8 (127.23.97.195:41225)
I20260812 11:11:25.682441 30461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:52505
I20260812 11:11:25.693260 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:25.724462 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.710451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.23.97.193:39063)
0812 11:11:25.710798 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:25.710812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:25.710836 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:11:25.710881 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:25.710885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:25.711059 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:25.711178 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:25.711807 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.712619 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.713415 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.713769 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.713853 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.716353 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:25.716361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:25.716422 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:25.716451 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:25.722655 (+ 6204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.723053 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.723057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.723059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.723143 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.723444 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.723446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.723447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.723686 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:25.723844 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:25.724036 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:11:25.724219 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:25.734504 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:25.727092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.23.97.254:41797)
0812 11:11:25.727239 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:25.727243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:25.727257 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:25.727325 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:25.727327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:25.727364 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:25.727467 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:25.727984 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.728524 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.729255 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:25.729457 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:25.729545 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:25.731642 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:25.731645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:25.731647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:25.731670 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:25.733035 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.733438 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.733441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.733442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.733511 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:25.733743 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:25.733745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:25.733747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:25.733934 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:25.734051 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:25.734310 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 11:11:25.734395 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 11:11:25.734864 30461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45604
I20260812 11:11:25.735459 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30563
I20260812 11:11:25.753504 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30792
I20260812 11:11:25.764462 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31025
I20260812 11:11:25.772360 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30411
2026-08-12T11:11:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2457 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[31291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[31291](info): set up 2 sockets
Aug 12 11:11:25 dist-test-slave-zpvx krb5kdc[31291](info): commencing operation
krb5kdc: starting...
W20260812 11:11:27.088356 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:27.211402 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:42395
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:27.320294 31312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:27.320588 31312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:27.320673 31312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:27.324558 31312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:27.324641 31312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:27.324679 31312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:27.324734 31312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:27.324771 31312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:27.329905 31312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:27.330888 31312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:27.331864 31312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:27.339901 31321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:27.339941 31325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:27.339941 31330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:27.340309 31312 server_base.cc:1034] running on GCE node
I20260812 11:11:27.340862 31312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:27.341910 31312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:27.343128 31312 hybrid_clock.cc:648] HybridClock initialized: now 1786533087343090 us; error 46 us; skew 500 ppm
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:27.345908 31312 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:27.346688 31312 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:27.346786 31312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:27.347020 31312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:27.349500 31312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "3027226a49cd4780a5fb2c8ec3a8d848"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "e813b34346fb02fc583459eec0200329"
server_key_iv: "d567d55a0ee2f3f90a7a1896a4bc9898"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.350090 31312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "3027226a49cd4780a5fb2c8ec3a8d848"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "e813b34346fb02fc583459eec0200329"
server_key_iv: "d567d55a0ee2f3f90a7a1896a4bc9898"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.353775 31312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:27.356077 31342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.357360 31312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:27.357502 31312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "3027226a49cd4780a5fb2c8ec3a8d848"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "e813b34346fb02fc583459eec0200329"
server_key_iv: "d567d55a0ee2f3f90a7a1896a4bc9898"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.357614 31312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:27.396394 31312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:27.399165 31312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:27.399379 31312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:27.406389 31312 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:42395
I20260812 11:11:27.406664 31434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:42395 every 8 connection(s)
I20260812 11:11:27.407073 31312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:27.409830 31435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:27.415261 31435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: Bootstrap starting.
I20260812 11:11:27.417309 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31312
I20260812 11:11:27.417452 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:27.417747 23943 external_mini_cluster.cc:1428] Setting key c23999696cd128d6721e73c4ea0a2903
I20260812 11:11:27.417941 31435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:27.418864 31435 log.cc:824] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:27.420948 31435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: No bootstrap required, opened a new log
I20260812 11:11:27.425208 31435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } }
I20260812 11:11:27.425503 31435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:27.425578 31435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3027226a49cd4780a5fb2c8ec3a8d848, State: Initialized, Role: FOLLOWER
I20260812 11:11:27.426086 31435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } }
I20260812 11:11:27.426218 31435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:27.426306 31435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:27.426411 31435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:27.427304 31435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } }
I20260812 11:11:27.427636 31435 leader_election.cc:302] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3027226a49cd4780a5fb2c8ec3a8d848; no voters:
I20260812 11:11:27.427976 31435 leader_election.cc:288] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:27.428084 31445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:27.428308 31445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [term 1 LEADER]: Becoming Leader. State: Replica: 3027226a49cd4780a5fb2c8ec3a8d848, State: Running, Role: LEADER
I20260812 11:11:27.428656 31445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } }
I20260812 11:11:27.429332 31435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:27.429797 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3027226a49cd4780a5fb2c8ec3a8d848. Latest consensus state: current_term: 1 leader_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } } }
I20260812 11:11:27.430202 31448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3027226a49cd4780a5fb2c8ec3a8d848" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42395 } } }
I20260812 11:11:27.430408 31448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:27.430835 31458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:27.430012 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:27.432880 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:27.418923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.23.97.254:42395)
0812 11:11:27.419372 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:27.419389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:27.419424 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:27.420032 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:27.420048 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:27.420355 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:27.420702 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:27.421599 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.422535 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.423724 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.424178 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.424297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:27.427220 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:27.427225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:27.427238 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:27.427264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:27.429447 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.429994 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.430000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.430006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.430172 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.430504 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.430510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.430511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.431013 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:27.431203 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:27.431614 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 11:11:27.431924 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
I20260812 11:11:27.434475 31458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:27.440604 31458 catalog_manager.cc:1347] Generated new cluster ID: 57d289bf2e8a4d9686816b01642ba634
I20260812 11:11:27.440701 31458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:27.450588 31458 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:27.451669 31458 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:27.462793 31458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: Generated new TSK 0
I20260812 11:11:27.463558 31458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:27.501147 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:27.613147 31473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:27.613421 31473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:27.613497 31473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:27.617525 31473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:27.617600 31473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:27.617692 31473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:27.622649 31473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:27.623588 31473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:27.624923 31473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:27.634298 31480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:27.634294 31484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:27.634464 31473 server_base.cc:1034] running on GCE node
W20260812 11:11:27.634644 31486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:27.635066 31473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:27.635638 31473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:27.636793 31473 hybrid_clock.cc:648] HybridClock initialized: now 1786533087636780 us; error 25 us; skew 500 ppm
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:27.639470 31473 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:27.640172 31473 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:27.640277 31473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:27.640540 31473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:27.642498 31473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "76e9d6297bef44f8ab2637c1c2fb7229"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "712d1442c3dafd91954446465afc89a9"
server_key_iv: "2f2bfe8e0403cfab2fc99163a82c11de"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.643051 31473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "76e9d6297bef44f8ab2637c1c2fb7229"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "712d1442c3dafd91954446465afc89a9"
server_key_iv: "2f2bfe8e0403cfab2fc99163a82c11de"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.646606 31473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:11:27.648873 31499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.650053 31473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:27.650184 31473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "76e9d6297bef44f8ab2637c1c2fb7229"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "712d1442c3dafd91954446465afc89a9"
server_key_iv: "2f2bfe8e0403cfab2fc99163a82c11de"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.650313 31473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:27.661908 31473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:27.664578 31473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:27.664768 31473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:27.665304 31473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:27.666101 31473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:27.666177 31473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.666250 31473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:27.666292 31473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.675261 31473 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:41233
I20260812 11:11:27.675321 31705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:41233 every 8 connection(s)
I20260812 11:11:27.675902 31473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:27.677523 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31473
I20260812 11:11:27.677671 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:27.678010 23943 external_mini_cluster.cc:1428] Setting key 5b073e68e9f0d7bbbf6e6c6c70d6a383
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:27.690680 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:27.677785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:43207 (local address 127.23.97.254:42395)
0812 11:11:27.677952 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:27.677958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:27.678983 (+ 1025us) server_negotiation.cc:407] Connection header received
0812 11:11:27.680723 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:27.680725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:27.680768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:27.680838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:27.682303 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.682831 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.683972 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.684410 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.684527 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:27.687060 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:27.687064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:27.687067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:27.687097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:27.688601 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.689145 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.689152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.689156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.689242 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.689584 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.689589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.689591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.689768 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:27.689889 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:27.690371 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 11:11:27.690494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:11:27.691417 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:27.678267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42395 (local address 127.23.97.193:43207)
0812 11:11:27.678850 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:27.678892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:27.680493 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:27.681006 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:27.681018 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:27.681564 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:27.682158 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:27.682172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.682961 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.682965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:27.683838 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:27.683847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.684584 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.684587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:27.684836 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:27.685432 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:27.685448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:27.686900 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:27.688731 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:27.688737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:27.688750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:27.689015 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:27.689337 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:27.689342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:27.689343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:27.689483 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:27.689889 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:27.689896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:27.690138 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 11:11:27.690325 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 11:11:27.690630 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:27.692781 31706 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42395
I20260812 11:11:27.693058 31706 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:27.693631 31706 heartbeater.cc:511] Master 127.23.97.254:42395 requested a full tablet report, sending...
I20260812 11:11:27.695470 31363 ts_manager.cc:194] Registered new tserver with Master: 76e9d6297bef44f8ab2637c1c2fb7229 (127.23.97.193:41233)
I20260812 11:11:27.696729 31363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:43207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:27.709226 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:27.821475 31718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:27.821710 31718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:27.821753 31718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:27.825696 31718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:27.825747 31718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:27.825803 31718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:27.830986 31718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:27.831961 31718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:27.832952 31718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:27.842046 31729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:27.842020 31732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:27.842088 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:27.842226 31718 server_base.cc:1034] running on GCE node
I20260812 11:11:27.842680 31718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:27.843278 31718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:27.844456 31718 hybrid_clock.cc:648] HybridClock initialized: now 1786533087844439 us; error 30 us; skew 500 ppm
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:27.847026 31718 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:27.847771 31718 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:27.847867 31718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:27.848101 31718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:27.850104 31718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "32e52454d57745fb8dcfd967b356466a"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "9635bb8f6b1337e5d69e7e27949219c3"
server_key_iv: "bdc181a9298a6c05576989a776220928"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.850658 31718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "32e52454d57745fb8dcfd967b356466a"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "9635bb8f6b1337e5d69e7e27949219c3"
server_key_iv: "bdc181a9298a6c05576989a776220928"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.854278 31718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:27.856758 31742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.857915 31718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:27.858047 31718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "32e52454d57745fb8dcfd967b356466a"
format_stamp: "Formatted at 2026-08-12 11:11:27 on dist-test-slave-zpvx"
server_key: "9635bb8f6b1337e5d69e7e27949219c3"
server_key_iv: "bdc181a9298a6c05576989a776220928"
server_key_version: "encryptionkey@0"
I20260812 11:11:27.858153 31718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:27.870698 31718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:27.873553 31718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:27.873760 31718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:27.874336 31718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:27.875140 31718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:27.875214 31718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.875275 31718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:27.875325 31718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:27.884284 31718 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:33989
I20260812 11:11:27.884322 31960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:33989 every 8 connection(s)
I20260812 11:11:27.884931 31718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:27.885149 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31718
I20260812 11:11:27.885291 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:27.885656 23943 external_mini_cluster.cc:1428] Setting key bc1f91a541391dcffcb4540dbeb833e9
Aug 12 11:11:27 dist-test-slave-zpvx krb5kdc[31291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:11:27.900769 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:27.886699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:59225 (local address 127.23.97.254:42395)
0812 11:11:27.886823 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:27.886827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:27.887957 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 11:11:27.889844 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:27.889848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:27.889903 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:27.890010 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:27.891406 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.891918 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.893804 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.894012 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.894097 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:27.896911 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:27.896914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:27.896916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:27.896941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:27.898439 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.899029 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.899039 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.899041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.899118 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:27.899518 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:27.899522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:27.899523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:27.899701 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:27.899820 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:27.900327 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 11:11:27.900501 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:11:27.901578 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:27.887162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42395 (local address 127.23.97.194:59225)
0812 11:11:27.887783 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:27.887843 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:27.889626 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:27.890195 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:27.890201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:27.890545 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:27.891238 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:27.891262 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.892061 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.892068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:27.893675 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:27.893690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:27.894189 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:27.894196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:27.894434 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:27.895057 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:27.895080 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:27.896735 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:27.898573 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:27.898579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:27.898592 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:27.898916 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:27.899275 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:27.899278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:27.899280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:27.899409 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:27.899832 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:27.899837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:27.900052 (+ 215us) client_negotiation.cc:769] Sending connection context
0812 11:11:27.900214 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 11:11:27.900561 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":151,"threads_started":1}
I20260812 11:11:27.902936 31961 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42395
I20260812 11:11:27.903195 31961 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:27.903633 31961 heartbeater.cc:511] Master 127.23.97.254:42395 requested a full tablet report, sending...
I20260812 11:11:27.904700 31363 ts_manager.cc:194] Registered new tserver with Master: 32e52454d57745fb8dcfd967b356466a (127.23.97.194:33989)
I20260812 11:11:27.905267 31363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:59225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:27.915179 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:28.028002 31970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:28.028291 31970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:28.028422 31970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:28.032756 31970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:28.032831 31970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:28.032917 31970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:28.038048 31970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:28.039157 31970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:28.040104 31970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:28.049459 31981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:28.049533 31977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:28.049459 31976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:28.049656 31970 server_base.cc:1034] running on GCE node
I20260812 11:11:28.050251 31970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:28.050872 31970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:28.052064 31970 hybrid_clock.cc:648] HybridClock initialized: now 1786533088052049 us; error 32 us; skew 500 ppm
Aug 12 11:11:28 dist-test-slave-zpvx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533088, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:28.055178 31970 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:28.055935 31970 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:28.056005 31970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:28.056170 31970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:28.058208 31970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "faf590af3001405e83d5e1fb93423d60"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "d26eb98ce7ca9bef36e23494d53e6c78"
server_key_iv: "d22e2e06c7c77686e8990658d6cbd366"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.058712 31970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "faf590af3001405e83d5e1fb93423d60"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "d26eb98ce7ca9bef36e23494d53e6c78"
server_key_iv: "d22e2e06c7c77686e8990658d6cbd366"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.062505 31970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:11:28.064818 31990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.066044 31970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:28.066141 31970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "faf590af3001405e83d5e1fb93423d60"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "d26eb98ce7ca9bef36e23494d53e6c78"
server_key_iv: "d22e2e06c7c77686e8990658d6cbd366"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.066275 31970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:28.081338 31970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:28.084277 31970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:28.084496 31970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:28.085057 31970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:28.085888 31970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:28.085935 31970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.085974 31970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:28.086036 31970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.095028 31970 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:34629
I20260812 11:11:28.095053 32192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:34629 every 8 connection(s)
I20260812 11:11:28.095578 31970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:28.101250 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31970
I20260812 11:11:28.101372 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:28.101645 23943 external_mini_cluster.cc:1428] Setting key f84493a6cde0b1c51cc81ebeff144652
Aug 12 11:11:28 dist-test-slave-zpvx krb5kdc[31291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533088, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:28.109632 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.097301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57207 (local address 127.23.97.254:42395)
0812 11:11:28.097465 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.097470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.098307 (+ 837us) server_negotiation.cc:407] Connection header received
0812 11:11:28.099594 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.099596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.099643 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.099718 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:28.101069 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.101855 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.103268 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.103594 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.103707 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.106110 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:28.106114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:28.106117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:28.106148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:28.107535 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:28.108043 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:28.108046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:28.108048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:28.108135 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:28.108542 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:28.108545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:28.108547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:28.108736 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:28.108846 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.109343 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.109464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:11:28.110344 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.097674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42395 (local address 127.23.97.195:57207)
0812 11:11:28.098153 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.098198 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.099396 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.099888 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.099894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.100275 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:28.100920 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.100936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.102032 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.102039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.103153 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.103164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.103757 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.103761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.103933 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.104553 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:28.104568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:28.105949 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:28.107655 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:28.107661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:28.107675 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:28.107930 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:28.108237 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:28.108240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:28.108242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:28.108434 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:28.108847 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:28.108854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:28.109098 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.109285 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.109582 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260812 11:11:28.111397 32193 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42395
I20260812 11:11:28.111642 32193 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:28.112085 32193 heartbeater.cc:511] Master 127.23.97.254:42395 requested a full tablet report, sending...
I20260812 11:11:28.113070 31363 ts_manager.cc:194] Registered new tserver with Master: faf590af3001405e83d5e1fb93423d60 (127.23.97.195:34629)
I20260812 11:11:28.113601 31363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:57207
I20260812 11:11:28.115693 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:11:28.124358 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.117614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.23.97.254:42395)
0812 11:11:28.117723 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.117727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.117933 (+ 206us) server_negotiation.cc:407] Connection header received
0812 11:11:28.118025 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.118027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.118061 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.118168 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:28.118832 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.119332 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.120085 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.120315 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.120446 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.121638 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:28.121641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:28.121643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:28.121665 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:28.122943 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:28.123337 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:28.123339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:28.123341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:28.123415 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:28.123651 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:28.123653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:28.123654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:28.123849 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:28.123954 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.124167 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.124255 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:11:28.127074 31363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:28.130551 31363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:28.145428 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.138979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.23.97.195:34629)
0812 11:11:28.140341 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.140354 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.140395 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:11:28.140457 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.140462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.140654 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.140820 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:28.141124 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142380 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.143906 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.144713 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.144828 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.144971 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.145177 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.145253 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":153,"threads_started":1}
I20260812 11:11:28.145437 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.138872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:34629 (local address 127.0.0.1:55684)
0812 11:11:28.139360 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.139405 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.139545 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.140820 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.140826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.140850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:28.140978 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.140988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142528 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.142532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.143804 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.143811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.144843 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.144847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.144970 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.145014 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.145153 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.145281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":81,"thread_start_us":77,"threads_started":1}
I20260812 11:11:28.145936 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.139825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.23.97.193:41233)
0812 11:11:28.140298 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.140310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.140335 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:11:28.140445 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.140451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.140654 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.140820 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:28.141363 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142077 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.144509 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.145257 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.145364 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.145503 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.145671 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.145776 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":117,"threads_started":1}
I20260812 11:11:28.146041 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.138971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:33989 (local address 127.0.0.1:56076)
0812 11:11:28.139941 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.139954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.140063 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.141058 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.141061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.141091 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:28.141196 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.141203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142592 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.142597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.143709 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.143719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.145626 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.145631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.145721 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.145742 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.145818 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.145886 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":66,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260812 11:11:28.146556 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.139475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41233 (local address 127.0.0.1:46256)
0812 11:11:28.140032 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.140045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.140141 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.140999 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.141001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.141044 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:28.141197 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.141203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142211 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.142216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.143789 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.143801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.145386 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.145390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.145499 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.145529 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.145629 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.145706 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":136,"spinlock_wait_cycles":4224,"thread_start_us":108,"threads_started":1}
I20260812 11:11:28.146663 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.139932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.23.97.194:33989)
0812 11:11:28.140582 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.140596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.140623 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:11:28.140704 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.140708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.140920 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.141085 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:28.141296 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.142432 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.143838 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.144712 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.144827 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.144971 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.146371 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.146480 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":104,"threads_started":1}
I20260812 11:11:28.149127 32069 tablet_service.cc:1467] Processing CreateTablet for tablet 48dae15a9f3340eab5d5450fc010f13e (DEFAULT_TABLE table=test-table [id=a35b10d0192547418e813c6ce84ec0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:28.149494 31835 tablet_service.cc:1467] Processing CreateTablet for tablet 48dae15a9f3340eab5d5450fc010f13e (DEFAULT_TABLE table=test-table [id=a35b10d0192547418e813c6ce84ec0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:28.150370 32069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dae15a9f3340eab5d5450fc010f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:28.150523 31835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dae15a9f3340eab5d5450fc010f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:28.150601 31581 tablet_service.cc:1467] Processing CreateTablet for tablet 48dae15a9f3340eab5d5450fc010f13e (DEFAULT_TABLE table=test-table [id=a35b10d0192547418e813c6ce84ec0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:28.151463 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dae15a9f3340eab5d5450fc010f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:28.157797 32216 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Bootstrap starting.
I20260812 11:11:28.158792 32219 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Bootstrap starting.
I20260812 11:11:28.160271 32216 tablet_bootstrap.cc:654] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:28.160573 32219 tablet_bootstrap.cc:654] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:28.161705 32216 log.cc:824] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:28.161912 32219 log.cc:824] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:28.162635 32218 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Bootstrap starting.
I20260812 11:11:28.163719 32219 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: No bootstrap required, opened a new log
I20260812 11:11:28.163869 32216 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: No bootstrap required, opened a new log
I20260812 11:11:28.163959 32219 ts_tablet_manager.cc:1397] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:11:28.164055 32216 ts_tablet_manager.cc:1397] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:28.164803 32218 tablet_bootstrap.cc:654] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:28.165995 32218 log.cc:824] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:28.167213 32219 raft_consensus.cc:348] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.167469 32219 raft_consensus.cc:374] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:28.167547 32219 raft_consensus.cc:729] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf590af3001405e83d5e1fb93423d60, State: Initialized, Role: FOLLOWER
I20260812 11:11:28.168027 32219 consensus_queue.cc:260] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.168092 32218 tablet_bootstrap.cc:492] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: No bootstrap required, opened a new log
I20260812 11:11:28.168301 32218 ts_tablet_manager.cc:1397] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:28.168731 32193 heartbeater.cc:503] Master 127.23.97.254:42395 was elected leader, sending a full tablet report...
I20260812 11:11:28.169121 32219 ts_tablet_manager.cc:1428] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:28.168941 32216 raft_consensus.cc:348] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.169200 32216 raft_consensus.cc:374] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:28.169253 32216 raft_consensus.cc:729] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e9d6297bef44f8ab2637c1c2fb7229, State: Initialized, Role: FOLLOWER
I20260812 11:11:28.169675 32216 consensus_queue.cc:260] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.170620 32216 ts_tablet_manager.cc:1428] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 11:11:28.171051 31706 heartbeater.cc:503] Master 127.23.97.254:42395 was elected leader, sending a full tablet report...
I20260812 11:11:28.172194 32218 raft_consensus.cc:348] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.172511 32218 raft_consensus.cc:374] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:28.172574 32218 raft_consensus.cc:729] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e52454d57745fb8dcfd967b356466a, State: Initialized, Role: FOLLOWER
I20260812 11:11:28.172981 32218 consensus_queue.cc:260] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.173733 31961 heartbeater.cc:503] Master 127.23.97.254:42395 was elected leader, sending a full tablet report...
I20260812 11:11:28.173977 32218 ts_tablet_manager.cc:1428] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260812 11:11:28.177075 31709 tablet.cc:2369] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:11:28.346482 32195 tablet.cc:2369] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:11:28.386165 31962 tablet.cc:2369] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:28.408032 32230 raft_consensus.cc:482] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:28.408284 32230 raft_consensus.cc:504] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.409643 32230 leader_election.cc:288] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e52454d57745fb8dcfd967b356466a (127.23.97.194:33989), 76e9d6297bef44f8ab2637c1c2fb7229 (127.23.97.193:41233)
I20260812 11:11:28.414315 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.410118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54417 (local address 127.23.97.193:41233)
0812 11:11:28.410316 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.410320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.410456 (+ 136us) server_negotiation.cc:407] Connection header received
0812 11:11:28.410652 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.410655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.410716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.410791 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:28.411186 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.411898 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413303 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.413713 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413846 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.413915 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.414114 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.414191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:11:28.414331 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.409970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41233 (local address 127.23.97.195:54417)
0812 11:11:28.410361 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.410375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.410529 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.410906 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.410914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.410930 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:28.411057 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.411065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.412028 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.412031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.413188 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.413196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.413843 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.413953 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.414004 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.414094 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.414169 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260812 11:11:28.414570 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.410428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57409 (local address 127.23.97.194:33989)
0812 11:11:28.410547 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.410551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.410561 (+ 10us) server_negotiation.cc:407] Connection header received
0812 11:11:28.410637 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.410640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.410689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.410777 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:28.411090 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.411625 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413142 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.413736 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.413898 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.414382 (+ 484us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.414456 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 11:11:28.414774 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.409938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:33989 (local address 127.23.97.195:57409)
0812 11:11:28.410253 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.410270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.410375 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.410813 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.410816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.410836 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:28.410957 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.410963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.411748 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.411754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.413009 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:28.413020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.413961 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.413964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:28.414132 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.414154 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.414231 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.414298 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":36}
I20260812 11:11:28.415032 31617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dae15a9f3340eab5d5450fc010f13e" candidate_uuid: "faf590af3001405e83d5e1fb93423d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" is_pre_election: true
I20260812 11:11:28.415382 31617 raft_consensus.cc:2454] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faf590af3001405e83d5e1fb93423d60 in term 0.
I20260812 11:11:28.415403 31878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dae15a9f3340eab5d5450fc010f13e" candidate_uuid: "faf590af3001405e83d5e1fb93423d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e52454d57745fb8dcfd967b356466a" is_pre_election: true
I20260812 11:11:28.415673 31878 raft_consensus.cc:2454] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faf590af3001405e83d5e1fb93423d60 in term 0.
I20260812 11:11:28.415916 31995 leader_election.cc:302] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e9d6297bef44f8ab2637c1c2fb7229, faf590af3001405e83d5e1fb93423d60; no voters:
I20260812 11:11:28.416198 32230 raft_consensus.cc:2781] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:28.416299 32230 raft_consensus.cc:482] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:28.416344 32230 raft_consensus.cc:3037] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:28.417685 32230 raft_consensus.cc:504] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.418045 32230 leader_election.cc:288] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [CANDIDATE]: Term 1 election: Requested vote from peers 32e52454d57745fb8dcfd967b356466a (127.23.97.194:33989), 76e9d6297bef44f8ab2637c1c2fb7229 (127.23.97.193:41233)
I20260812 11:11:28.418407 31878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dae15a9f3340eab5d5450fc010f13e" candidate_uuid: "faf590af3001405e83d5e1fb93423d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e52454d57745fb8dcfd967b356466a"
I20260812 11:11:28.418529 31878 raft_consensus.cc:3037] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:28.419749 31878 raft_consensus.cc:2454] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faf590af3001405e83d5e1fb93423d60 in term 1.
I20260812 11:11:28.420019 31617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dae15a9f3340eab5d5450fc010f13e" candidate_uuid: "faf590af3001405e83d5e1fb93423d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e9d6297bef44f8ab2637c1c2fb7229"
I20260812 11:11:28.420053 31991 leader_election.cc:302] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e52454d57745fb8dcfd967b356466a, faf590af3001405e83d5e1fb93423d60; no voters:
I20260812 11:11:28.420133 31617 raft_consensus.cc:3037] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:28.420281 32230 raft_consensus.cc:2781] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:28.420528 32230 raft_consensus.cc:686] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 1 LEADER]: Becoming Leader. State: Replica: faf590af3001405e83d5e1fb93423d60, State: Running, Role: LEADER
I20260812 11:11:28.420849 32230 consensus_queue.cc:237] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } }
I20260812 11:11:28.421334 31617 raft_consensus.cc:2454] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faf590af3001405e83d5e1fb93423d60 in term 1.
I20260812 11:11:28.423857 31363 catalog_manager.cc:5495] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to faf590af3001405e83d5e1fb93423d60 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "faf590af3001405e83d5e1fb93423d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf590af3001405e83d5e1fb93423d60" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:28.442669 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.438864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.23.97.195:34629)
0812 11:11:28.438988 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.438994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.439034 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:11:28.439278 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.439280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.439332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.439438 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:28.439713 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.440280 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.441187 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:28.441425 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:28.441539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:28.441860 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:28.442250 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:28.442383 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.442460 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.442512 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:11:28.448268 31617 raft_consensus.cc:1261] T 48dae15a9f3340eab5d5450fc010f13e P 76e9d6297bef44f8ab2637c1c2fb7229 [term 1 FOLLOWER]: Refusing update from remote peer faf590af3001405e83d5e1fb93423d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:28.448268 31878 raft_consensus.cc:1261] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 1 FOLLOWER]: Refusing update from remote peer faf590af3001405e83d5e1fb93423d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:28.449000 32230 consensus_queue.cc:1035] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e9d6297bef44f8ab2637c1c2fb7229" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:28.449177 32239 consensus_queue.cc:1035] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e52454d57745fb8dcfd967b356466a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:28.456730 32242 mvcc.cc:205] Tried to move back new op lower bound from 7317639530277089280 to 7317639530174865408. Current Snapshot: MvccSnapshot[applied={T|T < 7317639530277089280}]
I20260812 11:11:28.457028 32247 mvcc.cc:205] Tried to move back new op lower bound from 7317639530277089280 to 7317639530174865408. Current Snapshot: MvccSnapshot[applied={T|T < 7317639530277089280}]
I20260812 11:11:28.465269 31362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:28.465418 31362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:28.467852 31362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3027226a49cd4780a5fb2c8ec3a8d848: Sending DeleteTablet for 3 replicas of tablet 48dae15a9f3340eab5d5450fc010f13e
I20260812 11:11:28.468724 31835 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dae15a9f3340eab5d5450fc010f13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:28 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:56076
I20260812 11:11:28.468801 32069 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dae15a9f3340eab5d5450fc010f13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:28 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:55684
I20260812 11:11:28.469357 32262 tablet_replica.cc:330] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: stopping tablet replica
I20260812 11:11:28.469506 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31473
I20260812 11:11:28.469473 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dae15a9f3340eab5d5450fc010f13e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:28 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:46256
I20260812 11:11:28.470316 32263 tablet_replica.cc:330] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: stopping tablet replica
I20260812 11:11:28.470752 32263 raft_consensus.cc:2229] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:28.471292 32263 raft_consensus.cc:2258] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:28.473446 32262 raft_consensus.cc:2229] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:28.473829 32262 raft_consensus.cc:2258] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:28.480252 32262 ts_tablet_manager.cc:1905] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:28.482705 32262 ts_tablet_manager.cc:1918] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:28.482842 32262 log.cc:1196] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/48dae15a9f3340eab5d5450fc010f13e
I20260812 11:11:28.483263 32262 ts_tablet_manager.cc:1939] T 48dae15a9f3340eab5d5450fc010f13e P 32e52454d57745fb8dcfd967b356466a: Deleting consensus metadata
I20260812 11:11:28.484133 32263 ts_tablet_manager.cc:1905] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:28.484730 31345 catalog_manager.cc:4841] TS 32e52454d57745fb8dcfd967b356466a (127.23.97.194:33989): tablet 48dae15a9f3340eab5d5450fc010f13e (table test-table [id=a35b10d0192547418e813c6ce84ec0ae]) successfully deleted
W20260812 11:11:28.486430 31350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:41233 (error 108)
I20260812 11:11:28.487057 32263 ts_tablet_manager.cc:1918] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:28.487177 32263 log.cc:1196] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/48dae15a9f3340eab5d5450fc010f13e
I20260812 11:11:28.487558 32263 ts_tablet_manager.cc:1939] T 48dae15a9f3340eab5d5450fc010f13e P faf590af3001405e83d5e1fb93423d60: Deleting consensus metadata
I20260812 11:11:28.487584 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31718
I20260812 11:11:28.487618 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.487263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41233 (local address 127.0.0.1:46266)
0812 11:11:28.487418 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.487514 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:41233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 11:11:28.495477 31350 catalog_manager.cc:4584] TS 76e9d6297bef44f8ab2637c1c2fb7229 (127.23.97.193:41233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48dae15a9f3340eab5d5450fc010f13e: Network error: Client connection negotiation failed: client connection to 127.23.97.193:41233: connect: Connection refused (error 111)
I20260812 11:11:28.496866 31347 catalog_manager.cc:4841] TS faf590af3001405e83d5e1fb93423d60 (127.23.97.195:34629): tablet 48dae15a9f3340eab5d5450fc010f13e (table test-table [id=a35b10d0192547418e813c6ce84ec0ae]) successfully deleted
I20260812 11:11:28.502674 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31970
I20260812 11:11:28.524917 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31312
2026-08-12T11:11:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2750 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T11:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:28Z Disabled control of system clock
I20260812 11:11:28.561266 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:42961
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:11:28.670921 32272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:28.671228 32272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:28.671304 32272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:28.675369 32272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:28.675450 32272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:28.675479 32272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:28.675516 32272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:28.675554 32272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:28.680500 32272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:42961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:28.681602 32272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:28.683002 32272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:28.693718 32279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:28.693742 32282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:28.693953 32272 server_base.cc:1034] running on GCE node
W20260812 11:11:28.693718 32278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:28.694675 32272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:28.695822 32272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:28.697012 32272 hybrid_clock.cc:648] HybridClock initialized: now 1786533088696979 us; error 42 us; skew 500 ppm
I20260812 11:11:28.698868 32272 webserver.cc:460] Webserver started at http://127.23.97.254:36363/ using document root <none> and password file <none>
I20260812 11:11:28.699453 32272 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:28.699518 32272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:28.699812 32272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:28.701932 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "dc7526232ca24aa7bf2092e5c5a16629"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "3e94504fa40f9371933cf280c6b45f35"
server_key_iv: "060968541cf0931f1ee788db54cebbce"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.702507 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "dc7526232ca24aa7bf2092e5c5a16629"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "3e94504fa40f9371933cf280c6b45f35"
server_key_iv: "060968541cf0931f1ee788db54cebbce"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.706137 32272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:11:28.708563 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.709781 32272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:28.709915 32272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "dc7526232ca24aa7bf2092e5c5a16629"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "3e94504fa40f9371933cf280c6b45f35"
server_key_iv: "060968541cf0931f1ee788db54cebbce"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.710021 32272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:28.714557 32272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:28.715126 32272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:28.715312 32272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:28.722254 32272 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:42961
I20260812 11:11:28.722265 32378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:42961 every 8 connection(s)
I20260812 11:11:28.723106 32272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:28.726181 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:28.726753 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32272
I20260812 11:11:28.726886 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:28.727154 23943 external_mini_cluster.cc:1428] Setting key 14be7a658e25b95bb916d8aaec9e751f
I20260812 11:11:28.732545 32381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: Bootstrap starting.
I20260812 11:11:28.732844 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.728597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.23.97.254:42961)
0812 11:11:28.729278 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.729293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.729327 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:11:28.729942 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.729956 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.730293 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.730561 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:28.731455 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:28.731460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:28.731472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:28.731483 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:28.731540 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:28.731556 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:28.731678 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.731814 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.732102 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20260812 11:11:28.735025 32381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:28.735746 32381 log.cc:824] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:28.737886 32381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: No bootstrap required, opened a new log
I20260812 11:11:28.741444 32381 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } }
I20260812 11:11:28.741647 32381 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:28.741680 32381 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7526232ca24aa7bf2092e5c5a16629, State: Initialized, Role: FOLLOWER
I20260812 11:11:28.742080 32381 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } }
I20260812 11:11:28.742208 32381 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:28.742256 32381 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:28.742329 32381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:28.743098 32381 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } }
I20260812 11:11:28.743387 32381 leader_election.cc:302] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc7526232ca24aa7bf2092e5c5a16629; no voters:
I20260812 11:11:28.743625 32381 leader_election.cc:288] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:28.743763 32391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:28.744035 32391 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [term 1 LEADER]: Becoming Leader. State: Replica: dc7526232ca24aa7bf2092e5c5a16629, State: Running, Role: LEADER
I20260812 11:11:28.744354 32391 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } }
I20260812 11:11:28.744918 32381 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:28.746138 32397 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7526232ca24aa7bf2092e5c5a16629. Latest consensus state: current_term: 1 leader_uuid: "dc7526232ca24aa7bf2092e5c5a16629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } } }
I20260812 11:11:28.746280 32397 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:28.746577 32392 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc7526232ca24aa7bf2092e5c5a16629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7526232ca24aa7bf2092e5c5a16629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 42961 } } }
I20260812 11:11:28.746704 32392 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:28.746670 32407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:28.749748 32407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:28.754980 32407 catalog_manager.cc:1347] Generated new cluster ID: 2f7358a62aff4a9fb3c0af9a6253c176
I20260812 11:11:28.755062 32407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:28.770349 32407 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:28.770658 32407 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:28.785424 32407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: Generated new TSK 0
I20260812 11:11:28.786266 32407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:11:28.795766 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--builtin_ntp_servers=127.23.97.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:11:28.911695 32430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:28.911971 32430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:28.912055 32430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:28.916200 32430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:28.916277 32430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:28.916379 32430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:28.921077 32430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:28.922080 32430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:28.923416 32430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:28.932957 32440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:28.932968 32444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:28.933079 32430 server_base.cc:1034] running on GCE node
W20260812 11:11:28.932965 32438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:28.933610 32430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:28.934223 32430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:28.935415 32430 hybrid_clock.cc:648] HybridClock initialized: now 1786533088935404 us; error 26 us; skew 500 ppm
I20260812 11:11:28.939620 32430 webserver.cc:460] Webserver started at http://127.23.97.193:41703/ using document root <none> and password file <none>
I20260812 11:11:28.940200 32430 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:28.940264 32430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:28.940546 32430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:28.942574 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "d988e4af279243938d0f0b38348a1fbc"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "6d12daab8df093c38c522a8e8b7760b6"
server_key_iv: "9f059f40e5f7bd966ee02ababd1c14e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.943137 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "d988e4af279243938d0f0b38348a1fbc"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "6d12daab8df093c38c522a8e8b7760b6"
server_key_iv: "9f059f40e5f7bd966ee02ababd1c14e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.946990 32430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:11:28.949657 32454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.950876 32430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:28.951025 32430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "d988e4af279243938d0f0b38348a1fbc"
format_stamp: "Formatted at 2026-08-12 11:11:28 on dist-test-slave-zpvx"
server_key: "6d12daab8df093c38c522a8e8b7760b6"
server_key_iv: "9f059f40e5f7bd966ee02ababd1c14e1"
server_key_version: "encryptionkey@0"
I20260812 11:11:28.951134 32430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:28.955651 32430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:28.956245 32430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:28.956476 32430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:28.957026 32430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:28.957904 32430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:28.957952 32430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.958022 32430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:28.958055 32430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:28.966866 32430 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44445
I20260812 11:11:28.966881 32651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44445 every 8 connection(s)
I20260812 11:11:28.967800 32430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:28.973016 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32430
I20260812 11:11:28.973151 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:28.973506 23943 external_mini_cluster.cc:1428] Setting key 4738f081a7dab9e9a67800a4a15d4a9c
I20260812 11:11:28.973768 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.969547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:56497 (local address 127.23.97.254:42961)
0812 11:11:28.969738 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:28.969744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:28.970731 (+ 987us) server_negotiation.cc:407] Connection header received
0812 11:11:28.971980 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:28.971983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:28.972020 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:28.972095 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:28.973270 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:28.973278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:28.973280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:28.973293 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:28.973313 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:28.973321 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:28.973395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:28.973603 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 11:11:28.973655 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:11:28.974872 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:28.969942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42961 (local address 127.23.97.193:56497)
0812 11:11:28.970581 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:28.970620 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:28.971799 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:28.972274 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:28.972281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:28.972362 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:28.972907 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:28.972909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:28.972970 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:28.973059 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:28.973413 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:28.973424 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:28.973431 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:11:28.973616 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 11:11:28.973846 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260812 11:11:28.975634 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--builtin_ntp_servers=127.23.97.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:11:28.976101 32654 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42961
I20260812 11:11:28.976336 32654 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:28.976806 32654 heartbeater.cc:511] Master 127.23.97.254:42961 requested a full tablet report, sending...
I20260812 11:11:28.978469 32322 ts_manager.cc:194] Registered new tserver with Master: d988e4af279243938d0f0b38348a1fbc (127.23.97.193:44445)
W20260812 11:11:29.086930 32657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:29.087204 32657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:29.087275 32657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:29.091241 32657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:29.091321 32657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:29.091409 32657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:29.096190 32657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:29.097244 32657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:29.098596 32657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:29.108296 32669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:29.108415 32657 server_base.cc:1034] running on GCE node
W20260812 11:11:29.108312 32667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:29.108315 32671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:29.108965 32657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:29.109591 32657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:29.110761 32657 hybrid_clock.cc:648] HybridClock initialized: now 1786533089110745 us; error 27 us; skew 500 ppm
I20260812 11:11:29.112685 32657 webserver.cc:460] Webserver started at http://127.23.97.194:41287/ using document root <none> and password file <none>
I20260812 11:11:29.113338 32657 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:29.113404 32657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:29.113644 32657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:29.115700 32657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "d1c601e8daf44df8a32a234db4544e79"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "09d7644f68db02b2e6b04d65e6d92abc"
server_key_iv: "6f91b1f4f379a352fab30d873dff0beb"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.116271 32657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "d1c601e8daf44df8a32a234db4544e79"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "09d7644f68db02b2e6b04d65e6d92abc"
server_key_iv: "6f91b1f4f379a352fab30d873dff0beb"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.119763 32657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:29.122394 32682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.123661 32657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:29.123808 32657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "d1c601e8daf44df8a32a234db4544e79"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "09d7644f68db02b2e6b04d65e6d92abc"
server_key_iv: "6f91b1f4f379a352fab30d873dff0beb"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.123925 32657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:29.128332 32657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:29.128968 32657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:29.129168 32657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:29.129691 32657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:29.130581 32657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:29.130653 32657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.130720 32657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:29.130774 32657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.139688 32657 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:38359
I20260812 11:11:29.139719 420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:38359 every 8 connection(s)
I20260812 11:11:29.140647 32657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:29.142647 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32657
I20260812 11:11:29.142768 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:29.143085 23943 external_mini_cluster.cc:1428] Setting key 23fd4e6542f12898cc9a674fccf30096
I20260812 11:11:29.145197 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--builtin_ntp_servers=127.23.97.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:11:29.147428 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.142405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:41155 (local address 127.23.97.254:42961)
0812 11:11:29.142556 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.142560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.143901 (+ 1341us) server_negotiation.cc:407] Connection header received
0812 11:11:29.145557 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.145560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.145602 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.145705 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.146958 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.146964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.146969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.146987 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.147017 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.147028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.147116 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.147285 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.147324 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:11:29.148609 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.142931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42961 (local address 127.23.97.194:41155)
0812 11:11:29.143762 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.143803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.145323 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.145912 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.145919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.146003 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.146608 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.146610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.146650 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.146765 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.147123 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.147133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.147143 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.147322 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.147577 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":170,"threads_started":1}
I20260812 11:11:29.149742 421 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42961
I20260812 11:11:29.149984 421 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:29.150480 421 heartbeater.cc:511] Master 127.23.97.254:42961 requested a full tablet report, sending...
I20260812 11:11:29.151618 32322 ts_manager.cc:194] Registered new tserver with Master: d1c601e8daf44df8a32a234db4544e79 (127.23.97.194:38359)
W20260812 11:11:29.257336 425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:29.257568 425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:29.257606 425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:29.261471 425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:29.261518 425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:29.261569 425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:29.266162 425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:29.267115 425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:29.268478 425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:29.277766 440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:29.277763 436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:29.277763 433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:29.277810 425 server_base.cc:1034] running on GCE node
I20260812 11:11:29.278338 425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:29.278981 425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:29.280207 425 hybrid_clock.cc:648] HybridClock initialized: now 1786533089280191 us; error 30 us; skew 500 ppm
I20260812 11:11:29.282174 425 webserver.cc:460] Webserver started at http://127.23.97.195:44181/ using document root <none> and password file <none>
I20260812 11:11:29.282750 425 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:29.282845 425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:29.283090 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:29.285192 425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "dc15f5585c4c4f52b61613a6f8f33c53"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "9c2a89e4c17b186bb7f7c9e21dddacff"
server_key_iv: "3a5220ef2d3d07f1333770375e0f6f5b"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.285770 425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "dc15f5585c4c4f52b61613a6f8f33c53"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "9c2a89e4c17b186bb7f7c9e21dddacff"
server_key_iv: "3a5220ef2d3d07f1333770375e0f6f5b"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.289388 425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:29.291781 455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.293129 425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:29.293239 425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "dc15f5585c4c4f52b61613a6f8f33c53"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "9c2a89e4c17b186bb7f7c9e21dddacff"
server_key_iv: "3a5220ef2d3d07f1333770375e0f6f5b"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.293324 425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:29.297627 425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:29.298161 425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:29.298295 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:29.298759 425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:29.299582 425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:29.299633 425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.299669 425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:29.299692 425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.308143 425 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:38017
I20260812 11:11:29.308157 660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:38017 every 8 connection(s)
I20260812 11:11:29.309026 425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:29.312474 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 425
I20260812 11:11:29.312598 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:29.312923 23943 external_mini_cluster.cc:1428] Setting key b600a3ceeb5132419ddde3c837f786d5
I20260812 11:11:29.315493 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.310756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57225 (local address 127.23.97.254:42961)
0812 11:11:29.310899 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.310903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.311865 (+ 962us) server_negotiation.cc:407] Connection header received
0812 11:11:29.313606 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.313609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.313656 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.313765 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.314952 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.314955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.314958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.314970 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.314990 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.314999 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.315099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.315318 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.315364 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:29.316289 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.311114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:42961 (local address 127.23.97.195:57225)
0812 11:11:29.311723 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.311763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.313358 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.313952 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.313960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.314065 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.314702 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.314704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.314746 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.314809 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.315153 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.315162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.315175 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.315345 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.315571 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260812 11:11:29.317350 663 heartbeater.cc:347] Connected to a master server at 127.23.97.254:42961
I20260812 11:11:29.317567 663 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:29.317946 663 heartbeater.cc:511] Master 127.23.97.254:42961 requested a full tablet report, sending...
I20260812 11:11:29.318922 32322 ts_manager.cc:194] Registered new tserver with Master: dc15f5585c4c4f52b61613a6f8f33c53 (127.23.97.195:38017)
I20260812 11:11:29.326613 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:11:29.330185 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.328474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.23.97.254:42961)
0812 11:11:29.328604 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.328608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.328809 (+ 201us) server_negotiation.cc:407] Connection header received
0812 11:11:29.328954 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.328957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.328989 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.329091 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.329815 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.329818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.329821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.329832 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.329850 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.329858 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.329953 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.330029 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.330067 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:11:29.332763 32322 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:29.335661 32322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:29.347291 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.343780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:38359 (local address 127.0.0.1:53750)
0812 11:11:29.344163 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.344219 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.344325 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.345973 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.345975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.345989 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.346541 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.346543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.346552 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.346574 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.346892 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.346902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.346909 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.347044 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.347130 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260812 11:11:29.347325 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.343969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.23.97.194:38359)
0812 11:11:29.345588 (+ 1619us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.345601 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.345625 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:11:29.345674 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.345679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.345852 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.345989 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.346726 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.346730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.346739 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.346747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.346777 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.346793 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.346917 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.347071 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.347144 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":166,"threads_started":1}
I20260812 11:11:29.347512 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.344615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.23.97.195:38017)
0812 11:11:29.345101 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.345110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.345129 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:11:29.345170 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.345174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.345425 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.345567 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.346912 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.346918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.346927 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.346938 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.346975 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.346997 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.347102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.347303 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.347370 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":113,"threads_started":1}
I20260812 11:11:29.347625 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.344991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.23.97.193:44445)
0812 11:11:29.345379 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.345395 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.345417 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:11:29.345472 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.345480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.345639 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.346018 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.346912 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.346918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.346927 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.346938 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.346975 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.346997 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.347373 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.347416 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.347470 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":169,"threads_started":1}
I20260812 11:11:29.347767 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.344134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:38017 (local address 127.0.0.1:34716)
0812 11:11:29.344746 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.344759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.344840 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.345595 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.345603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.345678 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.346270 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.346272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.346293 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.346415 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.347139 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.347145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.347151 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.347218 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.347243 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":92,"threads_started":1}
I20260812 11:11:29.348067 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.344550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44445 (local address 127.0.0.1:38864)
0812 11:11:29.344823 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.344837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.344913 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.345789 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.345792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.345807 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.346380 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.346383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.346395 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.346429 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.347110 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.347115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.347122 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.347181 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.347220 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
I20260812 11:11:29.350518 549 tablet_service.cc:1467] Processing CreateTablet for tablet 76987c8736ce44a29c408d7a1bc86571 (DEFAULT_TABLE table=test-table [id=b82f49ea679240fdbf98ae2c494b8c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:29.350557 32544 tablet_service.cc:1467] Processing CreateTablet for tablet 76987c8736ce44a29c408d7a1bc86571 (DEFAULT_TABLE table=test-table [id=b82f49ea679240fdbf98ae2c494b8c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:29.351531 549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987c8736ce44a29c408d7a1bc86571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:29.351532 32544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987c8736ce44a29c408d7a1bc86571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:29.352826 32763 tablet_service.cc:1467] Processing CreateTablet for tablet 76987c8736ce44a29c408d7a1bc86571 (DEFAULT_TABLE table=test-table [id=b82f49ea679240fdbf98ae2c494b8c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:29.353822 32763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987c8736ce44a29c408d7a1bc86571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:29.358129 687 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: Bootstrap starting.
I20260812 11:11:29.360227 687 tablet_bootstrap.cc:654] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:29.360278 689 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Bootstrap starting.
I20260812 11:11:29.360870 688 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Bootstrap starting.
I20260812 11:11:29.361428 687 log.cc:824] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:29.362461 689 tablet_bootstrap.cc:654] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:29.362915 688 tablet_bootstrap.cc:654] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:29.363279 687 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: No bootstrap required, opened a new log
I20260812 11:11:29.363302 689 log.cc:824] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:29.363487 687 ts_tablet_manager.cc:1397] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:29.363719 688 log.cc:824] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:29.365090 689 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: No bootstrap required, opened a new log
I20260812 11:11:29.365305 689 ts_tablet_manager.cc:1397] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:11:29.365656 688 tablet_bootstrap.cc:492] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: No bootstrap required, opened a new log
I20260812 11:11:29.365845 688 ts_tablet_manager.cc:1397] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:11:29.368045 687 raft_consensus.cc:348] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.368320 687 raft_consensus.cc:374] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:29.368409 687 raft_consensus.cc:729] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d988e4af279243938d0f0b38348a1fbc, State: Initialized, Role: FOLLOWER
I20260812 11:11:29.368934 687 consensus_queue.cc:260] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.369007 689 raft_consensus.cc:348] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.369277 689 raft_consensus.cc:374] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:29.369350 689 raft_consensus.cc:729] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c601e8daf44df8a32a234db4544e79, State: Initialized, Role: FOLLOWER
I20260812 11:11:29.369764 32654 heartbeater.cc:503] Master 127.23.97.254:42961 was elected leader, sending a full tablet report...
I20260812 11:11:29.369804 689 consensus_queue.cc:260] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.369984 687 ts_tablet_manager.cc:1428] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260812 11:11:29.370278 688 raft_consensus.cc:348] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.370527 688 raft_consensus.cc:374] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:29.370545 421 heartbeater.cc:503] Master 127.23.97.254:42961 was elected leader, sending a full tablet report...
I20260812 11:11:29.370595 688 raft_consensus.cc:729] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc15f5585c4c4f52b61613a6f8f33c53, State: Initialized, Role: FOLLOWER
I20260812 11:11:29.370738 689 ts_tablet_manager.cc:1428] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:29.371030 688 consensus_queue.cc:260] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.371682 663 heartbeater.cc:503] Master 127.23.97.254:42961 was elected leader, sending a full tablet report...
I20260812 11:11:29.371986 688 ts_tablet_manager.cc:1428] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260812 11:11:29.391508 423 tablet.cc:2369] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:29.402470 699 raft_consensus.cc:482] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:29.402647 699 raft_consensus.cc:504] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.403630 699 leader_election.cc:288] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c601e8daf44df8a32a234db4544e79 (127.23.97.194:38359), d988e4af279243938d0f0b38348a1fbc (127.23.97.193:44445)
I20260812 11:11:29.405757 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.403919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:38359 (local address 127.23.97.195:55589)
0812 11:11:29.404078 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.404095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.404226 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.404636 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.404641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.404660 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.405188 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.405191 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.405206 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.405250 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.405512 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.405516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.405519 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.405598 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.405633 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 11:11:29.405757 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.404015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:55589 (local address 127.23.97.194:38359)
0812 11:11:29.404387 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.404393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.404407 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:29.404448 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.404451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.404490 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.404569 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.405373 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.405377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.405380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.405391 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.405420 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.405429 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.405525 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.405593 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.405633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260812 11:11:29.406225 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.404418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:34229 (local address 127.23.97.193:44445)
0812 11:11:29.404554 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.404560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.404758 (+ 198us) server_negotiation.cc:407] Connection header received
0812 11:11:29.404947 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.404951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.404990 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.405073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.405838 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.405841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.405844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.405856 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.405883 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.405891 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.405990 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.406057 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.406100 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:11:29.406281 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.404208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44445 (local address 127.23.97.195:34229)
0812 11:11:29.404656 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.404674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.404846 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.405103 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.405105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.405119 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:29.405657 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:11:29.405662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:29.405676 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:11:29.405703 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:29.405978 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:29.405981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:29.405984 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 11:11:29.406068 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:11:29.406100 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":152,"threads_started":1}
I20260812 11:11:29.406432 334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987c8736ce44a29c408d7a1bc86571" candidate_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c601e8daf44df8a32a234db4544e79" is_pre_election: true
I20260812 11:11:29.406733 334 raft_consensus.cc:2454] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc15f5585c4c4f52b61613a6f8f33c53 in term 0.
I20260812 11:11:29.406980 32582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987c8736ce44a29c408d7a1bc86571" candidate_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d988e4af279243938d0f0b38348a1fbc" is_pre_election: true
I20260812 11:11:29.407200 457 leader_election.cc:302] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c601e8daf44df8a32a234db4544e79, dc15f5585c4c4f52b61613a6f8f33c53; no voters:
I20260812 11:11:29.407256 32582 raft_consensus.cc:2454] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc15f5585c4c4f52b61613a6f8f33c53 in term 0.
I20260812 11:11:29.407503 699 raft_consensus.cc:2781] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:29.407629 699 raft_consensus.cc:482] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:29.407693 699 raft_consensus.cc:3037] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:29.408845 699 raft_consensus.cc:504] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.409190 699 leader_election.cc:288] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [CANDIDATE]: Term 1 election: Requested vote from peers d1c601e8daf44df8a32a234db4544e79 (127.23.97.194:38359), d988e4af279243938d0f0b38348a1fbc (127.23.97.193:44445)
I20260812 11:11:29.409598 334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987c8736ce44a29c408d7a1bc86571" candidate_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c601e8daf44df8a32a234db4544e79"
I20260812 11:11:29.409598 32582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987c8736ce44a29c408d7a1bc86571" candidate_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d988e4af279243938d0f0b38348a1fbc"
I20260812 11:11:29.409721 334 raft_consensus.cc:3037] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:29.409732 32582 raft_consensus.cc:3037] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:29.410989 334 raft_consensus.cc:2454] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc15f5585c4c4f52b61613a6f8f33c53 in term 1.
I20260812 11:11:29.410989 32582 raft_consensus.cc:2454] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc15f5585c4c4f52b61613a6f8f33c53 in term 1.
I20260812 11:11:29.411263 457 leader_election.cc:302] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c601e8daf44df8a32a234db4544e79, dc15f5585c4c4f52b61613a6f8f33c53; no voters:
I20260812 11:11:29.411468 699 raft_consensus.cc:2781] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:29.411716 699 raft_consensus.cc:686] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 1 LEADER]: Becoming Leader. State: Replica: dc15f5585c4c4f52b61613a6f8f33c53, State: Running, Role: LEADER
I20260812 11:11:29.412019 699 consensus_queue.cc:237] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } }
I20260812 11:11:29.415300 32322 catalog_manager.cc:5495] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc15f5585c4c4f52b61613a6f8f33c53 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc15f5585c4c4f52b61613a6f8f33c53" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38017 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:29.426409 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.424848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.23.97.195:38017)
0812 11:11:29.424975 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.424980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.424995 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:29.425111 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.425113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.425171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.425269 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.425988 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.425991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.425994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.426006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.426026 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.426043 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.426135 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.426218 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.426262 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:11:29.432068 32582 raft_consensus.cc:1261] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 1 FOLLOWER]: Refusing update from remote peer dc15f5585c4c4f52b61613a6f8f33c53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:29.432071 334 raft_consensus.cc:1261] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 1 FOLLOWER]: Refusing update from remote peer dc15f5585c4c4f52b61613a6f8f33c53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:29.432845 699 consensus_queue.cc:1035] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d988e4af279243938d0f0b38348a1fbc" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:29.433019 701 consensus_queue.cc:1035] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c601e8daf44df8a32a234db4544e79" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:29.439769 709 mvcc.cc:205] Tried to move back new op lower bound from 7317639534307274752 to 7317639534234370048. Current Snapshot: MvccSnapshot[applied={T|T < 7317639534307274752}]
I20260812 11:11:29.447566 32320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:29.447764 32320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:29.450186 32320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc7526232ca24aa7bf2092e5c5a16629: Sending DeleteTablet for 3 replicas of tablet 76987c8736ce44a29c408d7a1bc86571
I20260812 11:11:29.450894 32763 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987c8736ce44a29c408d7a1bc86571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:29 UTC) from {username='slave'} at 127.0.0.1:53750
I20260812 11:11:29.451022 32544 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987c8736ce44a29c408d7a1bc86571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:29 UTC) from {username='slave'} at 127.0.0.1:38864
I20260812 11:11:29.451438 727 tablet_replica.cc:330] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc: stopping tablet replica
I20260812 11:11:29.451452 549 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987c8736ce44a29c408d7a1bc86571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:29 UTC) from {username='slave'} at 127.0.0.1:34716
I20260812 11:11:29.451851 727 raft_consensus.cc:2229] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:29.451944 725 tablet_replica.cc:330] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: stopping tablet replica
I20260812 11:11:29.451990 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32430
I20260812 11:11:29.452320 727 raft_consensus.cc:2258] T 76987c8736ce44a29c408d7a1bc86571 P d988e4af279243938d0f0b38348a1fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:29.453243 725 raft_consensus.cc:2229] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:29.453598 725 raft_consensus.cc:2258] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:29.457021 725 ts_tablet_manager.cc:1905] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:29.461066 725 ts_tablet_manager.cc:1918] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:29.461210 725 log.cc:1196] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/76987c8736ce44a29c408d7a1bc86571
I20260812 11:11:29.461606 725 ts_tablet_manager.cc:1939] T 76987c8736ce44a29c408d7a1bc86571 P d1c601e8daf44df8a32a234db4544e79: Deleting consensus metadata
I20260812 11:11:29.462900 32295 catalog_manager.cc:4841] TS d1c601e8daf44df8a32a234db4544e79 (127.23.97.194:38359): tablet 76987c8736ce44a29c408d7a1bc86571 (table test-table [id=b82f49ea679240fdbf98ae2c494b8c2e]) successfully deleted
I20260812 11:11:29.466279 728 tablet_replica.cc:330] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: stopping tablet replica
I20260812 11:11:29.466624 728 raft_consensus.cc:2229] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:29.466992 728 raft_consensus.cc:2258] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:29.468920 728 ts_tablet_manager.cc:1905] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:29.472563 728 ts_tablet_manager.cc:1918] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:29.472687 728 log.cc:1196] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/76987c8736ce44a29c408d7a1bc86571
I20260812 11:11:29.473062 728 ts_tablet_manager.cc:1939] T 76987c8736ce44a29c408d7a1bc86571 P dc15f5585c4c4f52b61613a6f8f33c53: Deleting consensus metadata
W20260812 11:11:29.473301 32295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44445 (error 108)
I20260812 11:11:29.474195 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32657
I20260812 11:11:29.474484 32298 catalog_manager.cc:4841] TS dc15f5585c4c4f52b61613a6f8f33c53 (127.23.97.195:38017): tablet 76987c8736ce44a29c408d7a1bc86571 (table test-table [id=b82f49ea679240fdbf98ae2c494b8c2e]) successfully deleted
I20260812 11:11:29.482949 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.482604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44445 (local address 127.0.0.1:38870)
0812 11:11:29.482757 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.482842 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 11:11:29.483127 32295 catalog_manager.cc:4584] TS d988e4af279243938d0f0b38348a1fbc (127.23.97.193:44445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76987c8736ce44a29c408d7a1bc86571: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44445: connect: Connection refused (error 111)
I20260812 11:11:29.489683 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 425
I20260812 11:11:29.502038 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32272
2026-08-12T11:11:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (977 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 11:11:29.529883 23943 mini_oidc.cc:152] Starting JWKS server
I20260812 11:11:29.531661 23943 webserver.cc:460] Webserver started at https://127.0.0.1:36777/ using document root <none> and password file <none>
I20260812 11:11:29.531859 23943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:11:29.532145 23943 webserver.cc:460] Webserver started at http://127.0.0.1:40485/ using document root <none> and password file <none>
2026-08-12T11:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:29Z Disabled control of system clock
I20260812 11:11:29.542009 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40203
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36777/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/testchainca.cert with env {}
W20260812 11:11:29.652115 740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:29.652451 740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:29.652519 740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:29.656359 740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:29.656425 740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:29.656441 740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:29.656457 740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:29.656472 740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:11:29.656533 740 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:11:29.656553 740 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36777/jwks.json
I20260812 11:11:29.661792 740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36777/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:29.662835 740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:29.663848 740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:29.672542 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:29.672550 753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:29.672542 747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:29.672591 740 server_base.cc:1034] running on GCE node
I20260812 11:11:29.673295 740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:29.674374 740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:29.675565 740 hybrid_clock.cc:648] HybridClock initialized: now 1786533089675533 us; error 41 us; skew 500 ppm
I20260812 11:11:29.677520 740 webserver.cc:460] Webserver started at http://127.23.97.254:41223/ using document root <none> and password file <none>
I20260812 11:11:29.678097 740 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:29.678162 740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:29.678426 740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:29.680462 740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "9fd0e1e3f8f0428fbed761254b320661"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "11b64a0a9a83052e06278d11f4b762e1"
server_key_iv: "d7fa7d4b6aafeaa58f3e770ae76aadba"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.681110 740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "9fd0e1e3f8f0428fbed761254b320661"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "11b64a0a9a83052e06278d11f4b762e1"
server_key_iv: "d7fa7d4b6aafeaa58f3e770ae76aadba"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.684931 740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:11:29.687290 761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:29.688548 740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:29.688692 740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "9fd0e1e3f8f0428fbed761254b320661"
format_stamp: "Formatted at 2026-08-12 11:11:29 on dist-test-slave-zpvx"
server_key: "11b64a0a9a83052e06278d11f4b762e1"
server_key_iv: "d7fa7d4b6aafeaa58f3e770ae76aadba"
server_key_version: "encryptionkey@0"
I20260812 11:11:29.688807 740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:29.715149 740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:29.715829 740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:29.716027 740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:29.723026 740 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40203
I20260812 11:11:29.723037 849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40203 every 8 connection(s)
I20260812 11:11:29.723942 740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:29.727002 850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:29.727634 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 740
I20260812 11:11:29.727761 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:29.728071 23943 external_mini_cluster.cc:1428] Setting key 3b9c6020b0a92f042c0da73bde9d48cb
I20260812 11:11:29.733114 850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661: Bootstrap starting.
I20260812 11:11:29.735311 850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:29.736054 850 log.cc:824] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:29.737353 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.729516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.23.97.254:40203)
0812 11:11:29.730118 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.730132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.730167 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:29.730821 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.730836 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.731215 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.731548 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:29.731961 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.733025 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.733987 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.734451 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.734629 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:29.735507 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:29.735512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:29.735524 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:29.735894 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:29.735947 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:29.735966 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:29.736079 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:29.736207 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 11:11:29.736542 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260812 11:11:29.738169 850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661: No bootstrap required, opened a new log
I20260812 11:11:29.740049 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:11:29.742079 850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } }
I20260812 11:11:29.742283 850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:29.742359 850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd0e1e3f8f0428fbed761254b320661, State: Initialized, Role: FOLLOWER
I20260812 11:11:29.742801 850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } }
I20260812 11:11:29.742949 850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:29.743044 850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:29.743157 850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:29.743983 850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } }
I20260812 11:11:29.744338 850 leader_election.cc:302] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fd0e1e3f8f0428fbed761254b320661; no voters:
I20260812 11:11:29.744678 850 leader_election.cc:288] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:29.744766 860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:29.745044 860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd0e1e3f8f0428fbed761254b320661, State: Running, Role: LEADER
I20260812 11:11:29.745340 860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } }
I20260812 11:11:29.745852 850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:29.746869 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fd0e1e3f8f0428fbed761254b320661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } } }
I20260812 11:11:29.747020 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:29.747377 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fd0e1e3f8f0428fbed761254b320661. Latest consensus state: current_term: 1 leader_uuid: "9fd0e1e3f8f0428fbed761254b320661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd0e1e3f8f0428fbed761254b320661" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40203 } } }
I20260812 11:11:29.747447 879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:29.747478 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:29.750720 879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:29.756572 879 catalog_manager.cc:1347] Generated new cluster ID: 765cfd77400e43208f100fa1d280520b
I20260812 11:11:29.756645 879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:29.785825 879 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:29.786703 879 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:29.793145 879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fd0e1e3f8f0428fbed761254b320661: Generated new TSK 0
I20260812 11:11:29.793865 879 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:11:29.833305 913 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 11:11:29.833451 913 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:11:29.833895 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:29.809689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.23.97.254:40203)
0812 11:11:29.809835 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:29.809840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:29.810074 (+ 234us) server_negotiation.cc:407] Connection header received
0812 11:11:29.810279 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:29.810281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:29.810327 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:29.810412 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:11:29.810697 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.811199 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.812463 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.812688 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.812769 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:29.813141 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:11:29.833499 (+ 20358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:11:29.833787 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 11:11:29.833992 854 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 11:11:29.835395 912 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:29.809617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40203 (local address 127.0.0.1:58634)
0812 11:11:29.809983 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:29.809999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:29.810133 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:29.810454 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:29.810457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:29.810466 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:11:29.810588 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:29.810595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.811332 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.811336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:29.812326 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:29.812337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:29.812807 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:29.812811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:29.812910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:29.812964 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:11:29.833772 (+ 20808us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:11:29.833998 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.97.254:40203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.97.254:40203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 11:11:29.836261 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 740
2026-08-12T11:11:29Z chronyd exiting
I20260812 11:11:29.890226 23943 test_util.cc:175] -----------------------------------------------
I20260812 11:11:29.890336 23943 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533060973827-23943-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (362 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 11:11:29.891510 23943 mini_oidc.cc:152] Starting JWKS server
I20260812 11:11:29.892477 23943 webserver.cc:460] Webserver started at https://127.0.0.1:44267/ using document root <none> and password file <none>
I20260812 11:11:29.892606 23943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:11:29.892905 23943 webserver.cc:460] Webserver started at http://127.0.0.1:38969/ using document root <none> and password file <none>
2026-08-12T11:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:29Z Disabled control of system clock
I20260812 11:11:29.903033 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35425
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:35425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44267/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/testchainca.cert with env {}
W20260812 11:11:30.010798 925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:30.011034 925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:30.011071 925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:30.014884 925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:30.014930 925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:30.014946 925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:30.014963 925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:30.014978 925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:11:30.014993 925 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:11:30.015013 925 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44267/jwks.json
I20260812 11:11:30.020001 925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:35425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44267/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:30.020923 925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:30.021839 925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:30.030340 932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:30.030344 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:30.030467 925 server_base.cc:1034] running on GCE node
W20260812 11:11:30.030382 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:30.031222 925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:30.032265 925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:30.033469 925 hybrid_clock.cc:648] HybridClock initialized: now 1786533090033449 us; error 31 us; skew 500 ppm
I20260812 11:11:30.035363 925 webserver.cc:460] Webserver started at http://127.23.97.254:36487/ using document root <none> and password file <none>
I20260812 11:11:30.035965 925 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:30.036062 925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:30.036293 925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:30.038295 925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "a5372e0af178438c890bc50d5578830d"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "203c265592ae3cdfc3adcd36a7e93643"
server_key_iv: "54ff0a3bf6ae47486b5db673a786f3b4"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.038884 925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "a5372e0af178438c890bc50d5578830d"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "203c265592ae3cdfc3adcd36a7e93643"
server_key_iv: "54ff0a3bf6ae47486b5db673a786f3b4"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.042551 925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:11:30.045025 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:30.046332 925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:30.046478 925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "a5372e0af178438c890bc50d5578830d"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "203c265592ae3cdfc3adcd36a7e93643"
server_key_iv: "54ff0a3bf6ae47486b5db673a786f3b4"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.046591 925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:30.066924 925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:30.067567 925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:30.067780 925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:30.075146 925 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:35425
I20260812 11:11:30.075162 1038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:35425 every 8 connection(s)
I20260812 11:11:30.076107 925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:30.078465 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 925
I20260812 11:11:30.078596 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:30.078939 23943 external_mini_cluster.cc:1428] Setting key 0a160c7fb88416f5e987e71c8dc31c69
I20260812 11:11:30.079787 1046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:30.086017 1046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d: Bootstrap starting.
I20260812 11:11:30.088284 1046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:30.088297 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.080425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.23.97.254:35425)
0812 11:11:30.081089 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.081103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.081139 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:11:30.081924 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.081942 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.082338 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.082683 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:30.083113 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.084058 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.085040 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.085511 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.085685 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.086642 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:30.086647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:30.086659 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:30.087035 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:30.087088 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:30.087107 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:30.087220 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:30.087341 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 11:11:30.087597 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":152,"threads_started":1}
I20260812 11:11:30.089180 1046 log.cc:824] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:30.091127 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:11:30.091313 1046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d: No bootstrap required, opened a new log
I20260812 11:11:30.095667 1046 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } }
I20260812 11:11:30.095970 1046 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:30.096050 1046 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5372e0af178438c890bc50d5578830d, State: Initialized, Role: FOLLOWER
I20260812 11:11:30.096637 1046 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } }
I20260812 11:11:30.096822 1046 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:30.096900 1046 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:30.097005 1046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:30.097930 1046 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } }
I20260812 11:11:30.098315 1046 leader_election.cc:302] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5372e0af178438c890bc50d5578830d; no voters:
I20260812 11:11:30.098707 1046 leader_election.cc:288] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:30.098793 1058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:30.098994 1058 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [term 1 LEADER]: Becoming Leader. State: Replica: a5372e0af178438c890bc50d5578830d, State: Running, Role: LEADER
I20260812 11:11:30.099282 1058 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } }
I20260812 11:11:30.100028 1046 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:30.100765 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5372e0af178438c890bc50d5578830d. Latest consensus state: current_term: 1 leader_uuid: "a5372e0af178438c890bc50d5578830d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } } }
I20260812 11:11:30.100905 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.101334 1059 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5372e0af178438c890bc50d5578830d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5372e0af178438c890bc50d5578830d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35425 } } }
I20260812 11:11:30.101442 1059 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.102711 1064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:30.111158 1064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:30.117728 1064 catalog_manager.cc:1347] Generated new cluster ID: e37d608cefe7423eb7b2c36c52747093
I20260812 11:11:30.117811 1064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:30.134148 1064 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:30.135067 1064 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:30.140090 1064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5372e0af178438c890bc50d5578830d: Generated new TSK 0
I20260812 11:11:30.140841 1064 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:11:30.163025 1098 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 11:11:30.163130 1098 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:11:30.163837 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.147838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.23.97.254:35425)
0812 11:11:30.147989 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.147994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.148165 (+ 171us) server_negotiation.cc:407] Connection header received
0812 11:11:30.148288 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.148291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.148351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.148484 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:11:30.148713 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.149553 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.150477 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.150818 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.150924 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.151259 (+ 335us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:11:30.163430 (+ 12171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:11:30.163696 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 11:11:30.163904 1051 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 11:11:30.163968 1096 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:30.147717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35425 (local address 127.0.0.1:59554)
0812 11:11:30.148066 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:30.148078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:30.148177 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:30.148506 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:30.148509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:30.148516 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:11:30.148611 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.148616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.149678 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.149681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.150364 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.150374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.150942 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.150945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.151006 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.151029 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:11:30.163634 (+ 12605us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:11:30.163786 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.97.254:35425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
I20260812 11:11:30.164983 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 925
2026-08-12T11:11:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 11:11:30.221056 23943 mini_oidc.cc:152] Starting JWKS server
I20260812 11:11:30.221912 23943 webserver.cc:460] Webserver started at https://127.0.0.1:42979/ using document root <none> and password file <none>
I20260812 11:11:30.221997 23943 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:11:30.222208 23943 webserver.cc:460] Webserver started at http://127.0.0.1:34643/ using document root <none> and password file <none>
2026-08-12T11:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:30Z Disabled control of system clock
I20260812 11:11:30.232223 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43671
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42979/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:11:30.345602 1109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:30.345849 1109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:30.345885 1109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:30.349905 1109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:30.349958 1109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:30.349973 1109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:30.349990 1109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:30.350005 1109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:11:30.350021 1109 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:11:30.350034 1109 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42979/jwks.json
I20260812 11:11:30.354656 1109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42979/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:30.355603 1109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:30.356966 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:30.365963 1119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:30.366194 1109 server_base.cc:1034] running on GCE node
W20260812 11:11:30.366107 1122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:30.365967 1118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:30.366959 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:30.368108 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:30.369278 1109 hybrid_clock.cc:648] HybridClock initialized: now 1786533090369266 us; error 29 us; skew 500 ppm
I20260812 11:11:30.371089 1109 webserver.cc:460] Webserver started at http://127.23.97.254:44929/ using document root <none> and password file <none>
I20260812 11:11:30.371667 1109 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:30.371732 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:30.371896 1109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:30.373984 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "61c1977014b348a98acca471f5da597e"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "99c80828cf6b77ad5f8d958bcc693204"
server_key_iv: "19b4f80547266ceec60e953e53355e1e"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.374518 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "61c1977014b348a98acca471f5da597e"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "99c80828cf6b77ad5f8d958bcc693204"
server_key_iv: "19b4f80547266ceec60e953e53355e1e"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.378218 1109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260812 11:11:30.380574 1129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:30.381882 1109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260812 11:11:30.382019 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "61c1977014b348a98acca471f5da597e"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "99c80828cf6b77ad5f8d958bcc693204"
server_key_iv: "19b4f80547266ceec60e953e53355e1e"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.382148 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:30.398633 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:30.399292 1109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:30.399492 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:30.406682 1211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43671 every 8 connection(s)
I20260812 11:11:30.406679 1109 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43671
I20260812 11:11:30.407646 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:30.408068 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1109
I20260812 11:11:30.408209 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:30.408617 23943 external_mini_cluster.cc:1428] Setting key b3e22202e5415d8775a7bfa1e643182e
I20260812 11:11:30.411007 1212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:30.416409 1212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e: Bootstrap starting.
I20260812 11:11:30.417894 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.410032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.23.97.254:43671)
0812 11:11:30.410701 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.410717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.410753 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:11:30.411440 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.411458 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.411888 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.412271 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:30.412723 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.413670 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.414560 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.415021 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.415208 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.416092 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:30.416097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:30.416109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:30.416588 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:30.416657 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:30.416678 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:30.416789 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:30.416905 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 11:11:30.417166 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260812 11:11:30.418893 1212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:30.419688 1212 log.cc:824] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:30.420833 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:11:30.421868 1212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e: No bootstrap required, opened a new log
I20260812 11:11:30.425676 1212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } }
I20260812 11:11:30.425916 1212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:30.425997 1212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c1977014b348a98acca471f5da597e, State: Initialized, Role: FOLLOWER
I20260812 11:11:30.426414 1212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } }
I20260812 11:11:30.426563 1212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:30.426649 1212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:30.426771 1212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:30.427650 1212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } }
I20260812 11:11:30.427991 1212 leader_election.cc:302] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c1977014b348a98acca471f5da597e; no voters:
I20260812 11:11:30.428309 1212 leader_election.cc:288] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:30.428426 1219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:30.428699 1219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [term 1 LEADER]: Becoming Leader. State: Replica: 61c1977014b348a98acca471f5da597e, State: Running, Role: LEADER
I20260812 11:11:30.428992 1219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } }
I20260812 11:11:30.429654 1212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:30.430660 1222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c1977014b348a98acca471f5da597e. Latest consensus state: current_term: 1 leader_uuid: "61c1977014b348a98acca471f5da597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } } }
I20260812 11:11:30.430848 1222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.431217 1236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:30.431516 1221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c1977014b348a98acca471f5da597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c1977014b348a98acca471f5da597e" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43671 } } }
I20260812 11:11:30.431630 1221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.434455 1236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:30.439404 1236 catalog_manager.cc:1347] Generated new cluster ID: 1182c52ab64b4a498eff3395e668218b
I20260812 11:11:30.439463 1236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:30.447993 1236 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:30.449126 1236 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:30.459318 1236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c1977014b348a98acca471f5da597e: Generated new TSK 0
I20260812 11:11:30.459995 1236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:11:30.476715 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.471565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.23.97.254:43671)
0812 11:11:30.471708 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.471713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.471909 (+ 196us) server_negotiation.cc:407] Connection header received
0812 11:11:30.472068 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.472070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.472118 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.472205 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:11:30.472513 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.473041 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.474291 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.474511 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.474610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.474945 (+ 335us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:11:30.476336 (+ 1391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42979/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:11:30.476574 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42979/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 11:11:30.476790 1217 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42979/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 11:11:30.478571 1263 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:30.471505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43671 (local address 127.0.0.1:52322)
0812 11:11:30.471812 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:30.471828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:30.471946 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:30.472232 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:30.472236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:30.472244 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:11:30.472405 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.472413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.473149 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.473153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.474147 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.474158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.474639 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.474643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.474743 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.474771 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:11:30.476536 (+ 1765us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42979/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:11:30.478400 (+ 1864us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.97.254:43671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42979/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1665,"thread_start_us":116,"threads_started":1}
I20260812 11:11:30.479143 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1109
E20260812 11:11:30.499513 1265 webserver.cc:533] Webserver: sslize failed:
E20260812 11:11:30.499588 1265 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T11:11:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T11:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:30Z Disabled control of system clock
I20260812 11:11:30.540102 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:44241
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:44241
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/testchainca.cert with env {}
W20260812 11:11:30.649529 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:30.649767 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:30.649806 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:30.653645 1270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:30.653692 1270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:30.653707 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:30.653728 1270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:30.653743 1270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:11:30.653759 1270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:11:30.653779 1270 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 11:11:30.658794 1270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:44241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:30.659698 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:30.660679 1270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:11:30.668934 1270 server_base.cc:1034] running on GCE node
W20260812 11:11:30.668988 1278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:30.669098 1279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:30.668967 1282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:30.669776 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:30.670825 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:30.672019 1270 hybrid_clock.cc:648] HybridClock initialized: now 1786533090671993 us; error 37 us; skew 500 ppm
I20260812 11:11:30.673955 1270 webserver.cc:460] Webserver started at http://127.23.97.254:44533/ using document root <none> and password file <none>
I20260812 11:11:30.674541 1270 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:30.674639 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:30.674904 1270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:30.676971 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "bdb3602dfe6242b48305a4ad56507520"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "fd2c07de08d87ae436163c9dd94474c9"
server_key_iv: "67a48c00baadfd780246a6b6394d047b"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.677554 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "bdb3602dfe6242b48305a4ad56507520"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "fd2c07de08d87ae436163c9dd94474c9"
server_key_iv: "67a48c00baadfd780246a6b6394d047b"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.681102 1270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:11:30.683378 1297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:30.684600 1270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:30.684767 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "bdb3602dfe6242b48305a4ad56507520"
format_stamp: "Formatted at 2026-08-12 11:11:30 on dist-test-slave-zpvx"
server_key: "fd2c07de08d87ae436163c9dd94474c9"
server_key_iv: "67a48c00baadfd780246a6b6394d047b"
server_key_version: "encryptionkey@0"
I20260812 11:11:30.684899 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:30.699792 1270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:30.700484 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:30.700671 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:30.707491 1395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:44241 every 8 connection(s)
I20260812 11:11:30.707489 1270 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:44241
I20260812 11:11:30.708523 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:30.711259 1397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:30.715679 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1270
I20260812 11:11:30.715811 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:30.716135 23943 external_mini_cluster.cc:1428] Setting key d7062df422f250ce1c3c16b7f36e5ee3
I20260812 11:11:30.716928 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520: Bootstrap starting.
I20260812 11:11:30.719218 1397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:30.719944 1397 log.cc:824] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:30.721760 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520: No bootstrap required, opened a new log
I20260812 11:11:30.725090 1397 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } }
I20260812 11:11:30.725317 1397 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:30.725407 1397 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb3602dfe6242b48305a4ad56507520, State: Initialized, Role: FOLLOWER
I20260812 11:11:30.725864 1397 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } }
I20260812 11:11:30.726013 1397 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:30.726102 1397 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:30.726200 1397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:30.727010 1397 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } }
I20260812 11:11:30.727337 1397 leader_election.cc:302] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdb3602dfe6242b48305a4ad56507520; no voters:
I20260812 11:11:30.727658 1397 leader_election.cc:288] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:30.727887 1405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:30.728134 1405 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [term 1 LEADER]: Becoming Leader. State: Replica: bdb3602dfe6242b48305a4ad56507520, State: Running, Role: LEADER
I20260812 11:11:30.728520 1405 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } }
I20260812 11:11:30.728998 1397 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:30.730253 1407 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdb3602dfe6242b48305a4ad56507520. Latest consensus state: current_term: 1 leader_uuid: "bdb3602dfe6242b48305a4ad56507520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } } }
I20260812 11:11:30.730386 1407 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.730859 1406 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdb3602dfe6242b48305a4ad56507520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb3602dfe6242b48305a4ad56507520" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 44241 } } }
I20260812 11:11:30.730944 1406 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:30.731587 1423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:30.731287 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.717582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.23.97.254:44241)
0812 11:11:30.718224 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.718238 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.718273 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:11:30.718871 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.718889 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.719290 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.719640 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:30.720119 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.721115 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.722061 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.722524 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.722710 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.728890 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:30.728896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:30.728911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:11:30.729392 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:30.729462 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:11:30.729490 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:30.729600 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:30.729996 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 11:11:30.730292 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260812 11:11:30.734572 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:11:30.734804 1423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:30.739897 1423 catalog_manager.cc:1347] Generated new cluster ID: 0d00040cdff44811978d50cbe0d96139
I20260812 11:11:30.739977 1423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:30.768723 1423 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:30.769646 1423 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:30.774221 1423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdb3602dfe6242b48305a4ad56507520: Generated new TSK 0
I20260812 11:11:30.774868 1423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:11:30.840857 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:30.784091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.23.97.254:44241)
0812 11:11:30.784275 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:30.784280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:30.784572 (+ 292us) server_negotiation.cc:407] Connection header received
0812 11:11:30.784850 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:30.784854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:30.784945 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:30.785095 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:11:30.785362 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.786174 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.787054 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.787406 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.787503 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.787837 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:11:30.840325 (+ 52488us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:11:30.840691 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 11:11:30.840950 1402 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 11:11:30.841084 1449 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:30.783963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:44241 (local address 127.0.0.1:53752)
0812 11:11:30.784430 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:30.784457 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:30.784585 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:30.785101 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:30.785104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:30.785114 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:11:30.785236 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.785243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.786292 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.786295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.786939 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:30.786947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:30.787539 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:30.787542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:30.787611 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:30.787646 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:11:30.840639 (+ 52993us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:11:30.840850 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.97.254:44241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":117,"threads_started":1}
I20260812 11:11:30.841897 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1270
2026-08-12T11:11:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (343 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:31 dist-test-slave-zpvx krb5kdc[1473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:31 dist-test-slave-zpvx krb5kdc[1473](info): set up 2 sockets
Aug 12 11:11:31 dist-test-slave-zpvx krb5kdc[1473](info): commencing operation
krb5kdc: starting...
W20260812 11:11:32.509168 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.379s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:32 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:32.631166 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36625
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:32.746731 1502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:32.747037 1502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:32.747109 1502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:32.751139 1502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:32.751233 1502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:32.751273 1502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:32.751330 1502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:32.751370 1502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:32.756654 1502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:36625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:32.757726 1502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:32.758764 1502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:32.767176 1513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:32.767176 1515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:32.767243 1518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:32.767554 1502 server_base.cc:1034] running on GCE node
I20260812 11:11:32.768146 1502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:32.769297 1502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:32.770488 1502 hybrid_clock.cc:648] HybridClock initialized: now 1786533092770467 us; error 38 us; skew 500 ppm
Aug 12 11:11:32 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533092, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:32.773324 1502 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:32.774238 1502 webserver.cc:460] Webserver started at http://127.23.97.254:37319/ using document root <none> and password file <none>
I20260812 11:11:32.774765 1502 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:32.774823 1502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:32.774988 1502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:32.777069 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "3ef1778a4081405d96024a68f007dd67"
format_stamp: "Formatted at 2026-08-12 11:11:32 on dist-test-slave-zpvx"
server_key: "f4470160912f11c719fc8801311c6b6d"
server_key_iv: "dea6a4916c959397b26e87d8fcc4165a"
server_key_version: "encryptionkey@0"
I20260812 11:11:32.777599 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "3ef1778a4081405d96024a68f007dd67"
format_stamp: "Formatted at 2026-08-12 11:11:32 on dist-test-slave-zpvx"
server_key: "f4470160912f11c719fc8801311c6b6d"
server_key_iv: "dea6a4916c959397b26e87d8fcc4165a"
server_key_version: "encryptionkey@0"
I20260812 11:11:32.781375 1502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:11:32.783715 1531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:32.784989 1502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:32.785087 1502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "3ef1778a4081405d96024a68f007dd67"
format_stamp: "Formatted at 2026-08-12 11:11:32 on dist-test-slave-zpvx"
server_key: "f4470160912f11c719fc8801311c6b6d"
server_key_iv: "dea6a4916c959397b26e87d8fcc4165a"
server_key_version: "encryptionkey@0"
I20260812 11:11:32.785168 1502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:32.807601 1502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:32.810657 1502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:32.810824 1502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:32.818306 1502 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:36625
I20260812 11:11:32.818336 1620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:36625 every 8 connection(s)
I20260812 11:11:32.819340 1502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:32.822291 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:32.827718 1621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67: Bootstrap starting.
I20260812 11:11:32.828162 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1502
I20260812 11:11:32.828294 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:32.828614 23943 external_mini_cluster.cc:1428] Setting key de6d2b4abb053bed33d6a22b1b364147
I20260812 11:11:32.830581 1621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:32.831310 1621 log.cc:824] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:32.833474 1621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67: No bootstrap required, opened a new log
Aug 12 11:11:32 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:32.837925 1621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } }
I20260812 11:11:32.838188 1621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:32.838279 1621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef1778a4081405d96024a68f007dd67, State: Initialized, Role: FOLLOWER
I20260812 11:11:32.838766 1621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } }
I20260812 11:11:32.838940 1621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:32.839022 1621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:32.839138 1621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:32.840317 1621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } }
I20260812 11:11:32.840703 1621 leader_election.cc:302] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ef1778a4081405d96024a68f007dd67; no voters:
I20260812 11:11:32.841040 1621 leader_election.cc:288] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:32.841140 1631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:32.841377 1631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef1778a4081405d96024a68f007dd67, State: Running, Role: LEADER
I20260812 11:11:32.841745 1631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } }
I20260812 11:11:32.842253 1621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:32.843825 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ef1778a4081405d96024a68f007dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } } }
I20260812 11:11:32.844019 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:32.844197 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:32.830008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.23.97.254:36625)
0812 11:11:32.830457 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:32.830472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:32.830505 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:11:32.831217 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:32.831238 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:32.831607 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:32.831997 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:32.832904 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:32.833937 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:32.835109 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:32.835573 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:32.835694 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:32.838668 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:32.838675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:32.838690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:32.838721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:32.840869 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:32.841434 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:32.841442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:32.841448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:32.841587 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:32.841936 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:32.841939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:32.841941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:32.842409 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:32.842596 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:32.842971 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 11:11:32.843261 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260812 11:11:32.844873 1633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef1778a4081405d96024a68f007dd67. Latest consensus state: current_term: 1 leader_uuid: "3ef1778a4081405d96024a68f007dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef1778a4081405d96024a68f007dd67" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36625 } } }
I20260812 11:11:32.845081 1651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:32.845083 1633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:32.847884 1651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:32.853472 1651 catalog_manager.cc:1347] Generated new cluster ID: 9ceb6c0cf94e47cfbd1befa4d260c90c
I20260812 11:11:32.853539 1651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:32.861866 1651 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:32.862736 1651 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:32.875765 1651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ef1778a4081405d96024a68f007dd67: Generated new TSK 0
I20260812 11:11:32.876535 1651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:32.909288 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:33.021349 1673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:33.021595 1673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:33.021675 1673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:33.025638 1673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:33.025689 1673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:33.025780 1673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:33.030944 1673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:33.031970 1673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:33.033026 1673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:33.042747 1680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:33.042762 1682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.042800 1673 server_base.cc:1034] running on GCE node
W20260812 11:11:33.042730 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.043406 1673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:33.044018 1673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:33.045212 1673 hybrid_clock.cc:648] HybridClock initialized: now 1786533093045187 us; error 34 us; skew 500 ppm
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.048238 1673 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:33.049242 1673 webserver.cc:460] Webserver started at http://127.23.97.193:46653/ using document root <none> and password file <none>
I20260812 11:11:33.049847 1673 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:33.049947 1673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:33.050210 1673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:33.052419 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "07cd254893e74f7bab2a0c08ff74f87b"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "d3aa554b4206c8648f4bd975c0fccd69"
server_key_iv: "324e92e455e846b16c878f28afa50c2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.053054 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "07cd254893e74f7bab2a0c08ff74f87b"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "d3aa554b4206c8648f4bd975c0fccd69"
server_key_iv: "324e92e455e846b16c878f28afa50c2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.056922 1673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:11:33.059437 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.060783 1673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:33.060936 1673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "07cd254893e74f7bab2a0c08ff74f87b"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "d3aa554b4206c8648f4bd975c0fccd69"
server_key_iv: "324e92e455e846b16c878f28afa50c2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.061056 1673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:33.073907 1673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:33.076915 1673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:33.077126 1673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:33.077718 1673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:33.078575 1673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:33.078668 1673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.078747 1673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:33.078796 1673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.088930 1673 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:46549
I20260812 11:11:33.088951 1876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:46549 every 8 connection(s)
I20260812 11:11:33.090024 1673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:33.095443 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1673
I20260812 11:11:33.095592 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:33.095876 23943 external_mini_cluster.cc:1428] Setting key f9807f61682ce24ea561f35fead6e743
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:33.106244 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.091833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54687 (local address 127.23.97.254:36625)
0812 11:11:33.091977 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.091981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.092887 (+ 906us) server_negotiation.cc:407] Connection header received
0812 11:11:33.094462 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.094466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.094530 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.094647 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.095992 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.096744 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.098594 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.098850 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.098974 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.102144 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.102150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.102157 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.102191 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.103997 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.104636 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.104641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.104643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.104734 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.105097 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.105101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.105103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.105292 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.105404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.105930 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.106063 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:11:33.107036 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.092201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36625 (local address 127.23.97.193:54687)
0812 11:11:33.092734 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:33.092780 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:33.094216 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:33.094838 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:33.094845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:33.095231 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:33.095835 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.095853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.096890 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.096894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.098460 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.098471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.099017 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.099021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.099220 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.099827 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:33.099851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:33.101948 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:33.104146 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.104151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.104162 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.104513 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.104837 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.104841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.104843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.104984 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.105404 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:33.105412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:33.105664 (+ 252us) client_negotiation.cc:769] Sending connection context
0812 11:11:33.105857 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 11:11:33.106195 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:33.108395 1877 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36625
I20260812 11:11:33.108714 1877 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:33.109223 1877 heartbeater.cc:511] Master 127.23.97.254:36625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:33.110859 1558 ts_manager.cc:194] Registered new tserver with Master: 07cd254893e74f7bab2a0c08ff74f87b (127.23.97.193:46549)
I20260812 11:11:33.112018 1558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:54687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:33.127008 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:33.238596 1888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:33.238829 1888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:33.238870 1888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:33.243011 1888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:33.243067 1888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:33.243149 1888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:33.248212 1888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:33.249215 1888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:33.250168 1888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:33.259760 1898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:33.259794 1897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.259815 1888 server_base.cc:1034] running on GCE node
W20260812 11:11:33.259760 1903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.260567 1888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:33.261116 1888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:33.262300 1888 hybrid_clock.cc:648] HybridClock initialized: now 1786533093262279 us; error 30 us; skew 500 ppm
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.265201 1888 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:33.266168 1888 webserver.cc:460] Webserver started at http://127.23.97.194:36087/ using document root <none> and password file <none>
I20260812 11:11:33.266752 1888 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:33.266844 1888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:33.267066 1888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:33.269228 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "2eceb5aecea94938a6ee6fd5a78aea01"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "ca201c93cef94ad0ad1ad09cc4de4d22"
server_key_iv: "6ef2bb0af5de58b69a9cb40c92489ee0"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.269797 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "2eceb5aecea94938a6ee6fd5a78aea01"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "ca201c93cef94ad0ad1ad09cc4de4d22"
server_key_iv: "6ef2bb0af5de58b69a9cb40c92489ee0"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.273615 1888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 11:11:33.276160 1914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.277498 1888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:33.277652 1888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "2eceb5aecea94938a6ee6fd5a78aea01"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "ca201c93cef94ad0ad1ad09cc4de4d22"
server_key_iv: "6ef2bb0af5de58b69a9cb40c92489ee0"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.277771 1888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:33.301982 1888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:33.304911 1888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:33.305114 1888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:33.305706 1888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:33.306540 1888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:33.306613 1888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.306681 1888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:33.306727 1888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.315529 1888 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:41473
I20260812 11:11:33.315552 2109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:41473 every 8 connection(s)
I20260812 11:11:33.316490 1888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:33.323632 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1888
I20260812 11:11:33.323778 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:33.324090 23943 external_mini_cluster.cc:1428] Setting key e00a36b9e4d360fa8730fab6eef46708
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:33.331162 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.318355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:50649 (local address 127.23.97.254:36625)
0812 11:11:33.318519 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.318525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.319317 (+ 792us) server_negotiation.cc:407] Connection header received
0812 11:11:33.320725 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.320728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.320776 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.320860 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.322001 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.322515 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.323732 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.324013 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.324138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.327419 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.327426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.327433 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.327467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.329010 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.329571 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.329577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.329582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.329681 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.330010 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.330016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.330020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.330275 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.330405 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.330879 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.331004 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:11:33.331976 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.318714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36625 (local address 127.23.97.194:50649)
0812 11:11:33.319158 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:33.319200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:33.320521 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:33.321026 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:33.321033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:33.321379 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:33.321875 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.321889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.322645 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.322648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.323585 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.323596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.324169 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.324173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.324349 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.325448 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:33.325468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:33.327248 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:33.329150 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.329155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.329166 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.329453 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.329776 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.329778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.329779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.329912 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.330410 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:33.330416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:33.330637 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 11:11:33.330828 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 11:11:33.331162 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260812 11:11:33.333214 2111 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36625
I20260812 11:11:33.333459 2111 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:33.333940 2111 heartbeater.cc:511] Master 127.23.97.254:36625 requested a full tablet report, sending...
I20260812 11:11:33.335019 1558 ts_manager.cc:194] Registered new tserver with Master: 2eceb5aecea94938a6ee6fd5a78aea01 (127.23.97.194:41473)
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:11:33.335599 1558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:50649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:33.353531 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:33.466727 2124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:33.466992 2124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:33.467069 2124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:33.471010 2124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:33.471100 2124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:33.471185 2124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:33.476202 2124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:33.477195 2124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:33.478163 2124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:33.487772 2137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:33.487780 2132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.488018 2124 server_base.cc:1034] running on GCE node
W20260812 11:11:33.487804 2133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:33.488623 2124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:33.489287 2124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:33.490454 2124 hybrid_clock.cc:648] HybridClock initialized: now 1786533093490440 us; error 31 us; skew 500 ppm
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.493367 2124 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:33.494143 2124 webserver.cc:460] Webserver started at http://127.23.97.195:38279/ using document root <none> and password file <none>
I20260812 11:11:33.494663 2124 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:33.494721 2124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:33.494884 2124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:33.496879 2124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "547b16ad6aeb4a1493fe3ffbcd840ed5"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "52f10e57011b361bc2fa85af04c9a7b3"
server_key_iv: "7a074be61b1f4e9315bb5932228e2c46"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.497382 2124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "547b16ad6aeb4a1493fe3ffbcd840ed5"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "52f10e57011b361bc2fa85af04c9a7b3"
server_key_iv: "7a074be61b1f4e9315bb5932228e2c46"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.500952 2124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:33.503364 2150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.504557 2124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:33.504688 2124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "547b16ad6aeb4a1493fe3ffbcd840ed5"
format_stamp: "Formatted at 2026-08-12 11:11:33 on dist-test-slave-zpvx"
server_key: "52f10e57011b361bc2fa85af04c9a7b3"
server_key_iv: "7a074be61b1f4e9315bb5932228e2c46"
server_key_version: "encryptionkey@0"
I20260812 11:11:33.504786 2124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:33.515952 2124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:33.518612 2124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:33.518808 2124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:33.519364 2124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:33.520164 2124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:33.520212 2124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.520282 2124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:33.520332 2124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:33.528806 2124 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:42889
I20260812 11:11:33.528870 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:42889 every 8 connection(s)
I20260812 11:11:33.529691 2124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:33.539968 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 2124
I20260812 11:11:33.540112 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:33.540459 23943 external_mini_cluster.cc:1428] Setting key 78db247d2b311c31e8d0af852ee38d99
I20260812 11:11:33.542639 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.531387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:35283 (local address 127.23.97.254:36625)
0812 11:11:33.531547 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.531552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.532316 (+ 764us) server_negotiation.cc:407] Connection header received
0812 11:11:33.533699 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.533702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.533751 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.533826 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.534882 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.535397 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.536499 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.536706 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.536784 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.539037 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.539041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.539043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.539065 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.540479 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.541015 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.541018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.541020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.541112 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.541477 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.541479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.541481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.541643 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.541757 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.542337 (+ 580us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.542460 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:11:33.543454 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.531749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36625 (local address 127.23.97.195:35283)
0812 11:11:33.532169 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:33.532209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:33.533511 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:33.533983 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:33.533989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:33.534325 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:33.534759 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.534773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.535501 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.535504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.536348 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:33.536356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.536831 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.536834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:33.536968 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.537520 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:33.537538 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:33.538883 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:33.540613 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.540619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.540632 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.540936 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.541214 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:33.541216 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:33.541218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:33.541375 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:33.541759 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:33.541766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:33.542066 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 11:11:33.542269 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 11:11:33.542626 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260812 11:11:33.544621 2349 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36625
I20260812 11:11:33.544850 2349 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:33.545321 2349 heartbeater.cc:511] Master 127.23.97.254:36625 requested a full tablet report, sending...
I20260812 11:11:33.546259 1558 ts_manager.cc:194] Registered new tserver with Master: 547b16ad6aeb4a1493fe3ffbcd840ed5 (127.23.97.195:42889)
I20260812 11:11:33.546811 1558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:35283
I20260812 11:11:33.554169 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/test-admin.keytab.
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.572122 23943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 11:11:33.574877 2359 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:33.573538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.23.97.193:46549)
0812 11:11:33.573824 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.573839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.573864 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:11:33.573913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.573917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.574090 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.574208 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.574654 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33316: BlockingRecv error: recv got EOF from 127.0.0.1:33316 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
W20260812 11:11:33.574958 23943 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.576020 23943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.576973 23943 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:33.586225 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.577777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.23.97.193:46549)
0812 11:11:33.577918 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.577921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.577938 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:11:33.578049 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.578051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.578086 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.578151 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.578648 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.579396 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.580163 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.580576 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.580656 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.582912 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.582921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.582934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.582957 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.584652 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.585026 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.585031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.585032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.585107 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.585344 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.585346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.585348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.585570 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.585716 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.585880 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.586053 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 11:11:33.589956 2359 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:33.589037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.23.97.193:46549)
0812 11:11:33.589168 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.589171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.589202 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:11:33.589318 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.589321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.589363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.589423 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.589864 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33330: BlockingRecv error: recv got EOF from 127.0.0.1:33330 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 11:11:33.590063 23943 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.591141 23943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.592039 23943 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:33.600430 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.592885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.23.97.193:46549)
0812 11:11:33.593006 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.593011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.593115 (+ 104us) server_negotiation.cc:407] Connection header received
0812 11:11:33.593255 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.593258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.593308 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.593392 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.593889 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.594391 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.595175 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.595405 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.595514 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.597606 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.597615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.597618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.597639 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.598808 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.599223 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.599227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.599228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.599312 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.599602 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.599605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.599607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.599759 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.599910 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.600142 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.600256 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 11:11:33.602869 2359 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:33.601972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.23.97.193:46549)
0812 11:11:33.602094 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.602097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.602124 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:11:33.602226 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.602228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.602259 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.602322 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.602790 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33350: BlockingRecv error: recv got EOF from 127.0.0.1:33350 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 11:11:33.603003 23943 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533060973827-23943-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.603964 23943 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:33.604925 23943 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:11:33.612782 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:33.605712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.23.97.193:46549)
0812 11:11:33.605839 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:33.605843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:33.605910 (+ 67us) server_negotiation.cc:407] Connection header received
0812 11:11:33.606034 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:33.606037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:33.606080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:33.606164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:33.606645 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.607127 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.607925 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:33.608139 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:33.608248 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:33.610322 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:33.610326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:33.610327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:33.610348 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:33.611486 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.611848 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.611850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.611852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.611920 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:33.612141 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:33.612143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:33.612144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:33.612282 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:33.612476 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:33.612601 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 11:11:33.612685 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:11:33.613679 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1673
I20260812 11:11:33.631898 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1888
I20260812 11:11:33.639346 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 2124
I20260812 11:11:33.663551 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1502
2026-08-12T11:11:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2565 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[2385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[2385](info): set up 2 sockets
Aug 12 11:11:33 dist-test-slave-zpvx krb5kdc[2385](info): commencing operation
krb5kdc: starting...
W20260812 11:11:34.982780 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab.
Entry for principal oryx/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab.
I20260812 11:11:35.107925 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:45285
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab
--principal=oryx/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:35.221182 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:35.221467 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:35.221536 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:35.225414 2410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:35.225497 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:35.225533 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:35.225584 2410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:35.225636 2410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:35.230715 2410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab
--principal=oryx/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:35.231685 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:35.232679 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:35.240736 2417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:35.240806 2418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:35.240747 2421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:35.241072 2410 server_base.cc:1034] running on GCE node
I20260812 11:11:35.241689 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:35.242694 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:35.243893 2410 hybrid_clock.cc:648] HybridClock initialized: now 1786533095243863 us; error 51 us; skew 500 ppm
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:35.246665 2410 init.cc:377] Logged in from keytab as oryx/127.23.97.254@KRBTEST.COM (short username oryx)
I20260812 11:11:35.247484 2410 webserver.cc:460] Webserver started at http://127.23.97.254:42023/ using document root <none> and password file <none>
I20260812 11:11:35.248029 2410 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:35.248121 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:35.248318 2410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:35.250278 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "59c8040e0cf149b3b173902ea889d681"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "73710176a6ac60d0ab3d7d10dea764a5"
server_key_iv: "44e17f500a0d54f0f981afced07357c7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.250838 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "59c8040e0cf149b3b173902ea889d681"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "73710176a6ac60d0ab3d7d10dea764a5"
server_key_iv: "44e17f500a0d54f0f981afced07357c7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.254509 2410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:35.256832 2429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.258011 2410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:35.258150 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "59c8040e0cf149b3b173902ea889d681"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "73710176a6ac60d0ab3d7d10dea764a5"
server_key_iv: "44e17f500a0d54f0f981afced07357c7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.258256 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:35.285027 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:35.288025 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:35.288236 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:35.295539 2410 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:45285
I20260812 11:11:35.295841 2532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:45285 every 8 connection(s)
I20260812 11:11:35.296626 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:35.299358 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:35.303978 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 2410
I20260812 11:11:35.304140 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:35.304473 23943 external_mini_cluster.cc:1428] Setting key 595b2b5c8c864afa8117573af48d4e8f
I20260812 11:11:35.305099 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Bootstrap starting.
I20260812 11:11:35.307767 2536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:35.308697 2536 log.cc:824] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:35.310757 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: No bootstrap required, opened a new log
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:35.315057 2536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } }
I20260812 11:11:35.315305 2536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:35.315387 2536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c8040e0cf149b3b173902ea889d681, State: Initialized, Role: FOLLOWER
I20260812 11:11:35.315897 2536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } }
I20260812 11:11:35.316061 2536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:35.316150 2536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:35.316278 2536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:35.317157 2536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } }
I20260812 11:11:35.317504 2536 leader_election.cc:302] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c8040e0cf149b3b173902ea889d681; no voters:
I20260812 11:11:35.317836 2536 leader_election.cc:288] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:35.317896 2544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:35.318116 2544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 LEADER]: Becoming Leader. State: Replica: 59c8040e0cf149b3b173902ea889d681, State: Running, Role: LEADER
I20260812 11:11:35.318413 2544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } }
I20260812 11:11:35.319114 2536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:35.320245 2548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } }
I20260812 11:11:35.320245 2549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c8040e0cf149b3b173902ea889d681. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } }
I20260812 11:11:35.320349 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:35.305944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.23.97.254:45285)
0812 11:11:35.306389 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:35.306408 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:35.306451 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:11:35.307282 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:35.307306 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:35.307720 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:35.308122 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:35.308936 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.309977 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.311131 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.311597 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.311714 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:35.314556 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:35.314561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:35.314575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:35.314601 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:35.316650 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.317102 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.317108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.317114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.317233 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.317579 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.317582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.317584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.318093 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:35.318328 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:35.319055 (+ 727us) server_negotiation.cc:299] Negotiation successful
0812 11:11:35.319358 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260812 11:11:35.320411 2548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:35.320446 2549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:35.320815 2565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:35.325110 2565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:35.330554 2565 catalog_manager.cc:1347] Generated new cluster ID: 4cf482f93e894d95864228abacc3e8dc
I20260812 11:11:35.330614 2565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:35.346081 2565 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:35.347261 2565 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:35.361721 2565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Generated new TSK 0
I20260812 11:11:35.362550 2565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
I20260812 11:11:35.399288 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:35.512348 2587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:35.512593 2587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:35.512631 2587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:35.516690 2587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:35.516742 2587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:35.516863 2587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:35.522159 2587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:35.523137 2587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:35.524168 2587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:35.533679 2596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:35.533892 2587 server_base.cc:1034] running on GCE node
W20260812 11:11:35.533687 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:35.533699 2595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:35.534379 2587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:35.535036 2587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:35.536214 2587 hybrid_clock.cc:648] HybridClock initialized: now 1786533095536201 us; error 26 us; skew 500 ppm
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:35.542515 2587 init.cc:377] Logged in from keytab as oryx/127.23.97.193@KRBTEST.COM (short username oryx)
I20260812 11:11:35.543311 2587 webserver.cc:460] Webserver started at http://127.23.97.193:43213/ using document root <none> and password file <none>
I20260812 11:11:35.543808 2587 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:35.543865 2587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:35.544024 2587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:35.546047 2587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "72202d2e0bb842cf820d756339f6ac31"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "9349f1c6b34a664c33e47c5f603b3b72"
server_key_iv: "7013a1f3cb626fe6ae89d6e84f27dff7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.546555 2587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "72202d2e0bb842cf820d756339f6ac31"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "9349f1c6b34a664c33e47c5f603b3b72"
server_key_iv: "7013a1f3cb626fe6ae89d6e84f27dff7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.550222 2587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:11:35.552634 2609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.553938 2587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:35.554111 2587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "72202d2e0bb842cf820d756339f6ac31"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "9349f1c6b34a664c33e47c5f603b3b72"
server_key_iv: "7013a1f3cb626fe6ae89d6e84f27dff7"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.554231 2587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:35.572340 2587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:35.575280 2587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:35.575486 2587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:35.576078 2587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:35.576922 2587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:35.576997 2587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.577067 2587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:35.577126 2587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.585999 2587 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33243
I20260812 11:11:35.586038 2817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33243 every 8 connection(s)
I20260812 11:11:35.586851 2587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:35.595649 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 2587
I20260812 11:11:35.595826 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:35.596112 23943 external_mini_cluster.cc:1428] Setting key b963dbec99604c6619ce56754a111158
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:35.601058 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:35.588673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:60739 (local address 127.23.97.254:45285)
0812 11:11:35.588821 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:35.588825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:35.589668 (+ 843us) server_negotiation.cc:407] Connection header received
0812 11:11:35.591036 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:35.591039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:35.591119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:35.591200 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:35.592354 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.592917 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.594054 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.594319 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.594410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:35.597416 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:35.597419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:35.597421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:35.597445 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:35.598919 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.599459 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.599465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.599466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.599551 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.599906 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.599911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.599913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.600092 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:35.600199 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:35.600764 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 11:11:35.600886 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:35.602077 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:35.589060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:60739)
0812 11:11:35.589513 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:35.589555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:35.590821 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:35.591373 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:35.591379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:35.591723 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:35.592194 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:35.592208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.593041 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.593044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:35.593929 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:35.593938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.594491 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.594494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:35.594672 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:35.595334 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:35.595351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:35.597232 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:35.599051 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:35.599057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:35.599071 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:35.599349 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:35.599655 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:35.599659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:35.599661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:35.599806 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:35.600214 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:35.600222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:35.600520 (+ 298us) client_negotiation.cc:769] Sending connection context
0812 11:11:35.600705 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 11:11:35.601020 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260812 11:11:35.603415 2818 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:35.603690 2818 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:35.604193 2818 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:35.605841 2460 ts_manager.cc:194] Registered new tserver with Master: 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:35.607009 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:60739
WARNING: no policy specified for oryx/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
I20260812 11:11:35.626915 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:35.740840 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:35.741083 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:35.741151 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:35.745175 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:35.745235 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:35.745332 2826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:35.750466 2826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:35.751502 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:35.752556 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:35.762179 2842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:35.762243 2826 server_base.cc:1034] running on GCE node
W20260812 11:11:35.762195 2844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:35.762192 2836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:35.762822 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:35.763463 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:35.764664 2826 hybrid_clock.cc:648] HybridClock initialized: now 1786533095764623 us; error 48 us; skew 500 ppm
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:35.770893 2826 init.cc:377] Logged in from keytab as oryx/127.23.97.194@KRBTEST.COM (short username oryx)
I20260812 11:11:35.771867 2826 webserver.cc:460] Webserver started at http://127.23.97.194:37091/ using document root <none> and password file <none>
I20260812 11:11:35.772547 2826 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:35.772653 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:35.772900 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:35.774971 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "d996923ae74d4b2cbc93ce077273bfe1"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "e6b70a909d85a58f745e10be05bd9b2e"
server_key_iv: "5366eb7d428cd81d8d82f3bf02153205"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.775561 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "d996923ae74d4b2cbc93ce077273bfe1"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "e6b70a909d85a58f745e10be05bd9b2e"
server_key_iv: "5366eb7d428cd81d8d82f3bf02153205"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.779366 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 11:11:35.781884 2851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.783174 2826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:35.783327 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "d996923ae74d4b2cbc93ce077273bfe1"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "e6b70a909d85a58f745e10be05bd9b2e"
server_key_iv: "5366eb7d428cd81d8d82f3bf02153205"
server_key_version: "encryptionkey@0"
I20260812 11:11:35.783444 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:35.816586 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:35.819594 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:35.819823 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:35.820508 2826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:35.821377 2826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:35.821491 2826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.821557 2826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:35.821581 2826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:35.830318 2826 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:42505
I20260812 11:11:35.830344 3069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:42505 every 8 connection(s)
I20260812 11:11:35.831214 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:35.833952 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 2826
I20260812 11:11:35.834074 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:35.834349 23943 external_mini_cluster.cc:1428] Setting key cc9d20bab7af8fa55e743a942f97b104
Aug 12 11:11:35 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:11:35.845742 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:35.833003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:55113 (local address 127.23.97.254:45285)
0812 11:11:35.833130 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:35.833133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:35.833913 (+ 780us) server_negotiation.cc:407] Connection header received
0812 11:11:35.835559 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:35.835563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:35.835611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:35.835705 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:35.837301 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.837865 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.839002 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.839241 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.839354 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:35.842105 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:35.842108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:35.842110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:35.842134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:35.843595 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.844106 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.844110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.844112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.844197 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:35.844613 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:35.844616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:35.844618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:35.844800 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:35.844911 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:35.845450 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 11:11:35.845571 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:11:35.846772 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:35.833316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:55113)
0812 11:11:35.833768 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:35.833810 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:35.835310 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:35.835929 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:35.835938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:35.836471 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:35.837154 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:35.837176 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.838000 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.838003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:35.838871 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:35.838881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:35.839390 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:35.839394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:35.839630 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:35.840213 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:35.840237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:35.841933 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:35.843714 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:35.843720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:35.843741 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:35.843998 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:35.844300 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:35.844304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:35.844306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:35.844514 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:35.844909 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:35.844916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:35.845195 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 11:11:35.845394 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 11:11:35.845701 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260812 11:11:35.848089 3070 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:35.848450 3070 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.195@KRBTEST.COM" created.
I20260812 11:11:35.849112 3070 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:35.850407 2460 ts_manager.cc:194] Registered new tserver with Master: d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505)
I20260812 11:11:35.851277 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:55113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
I20260812 11:11:35.869529 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:35.993251 3078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:35.993521 3078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:35.993597 3078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:35.997743 3078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:35.997826 3078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:35.997915 3078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:36.002907 3078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:36.003907 3078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:36.004930 3078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:36.014856 3086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.014859 3094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.015069 3078 server_base.cc:1034] running on GCE node
W20260812 11:11:36.014873 3089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.015496 3078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.016112 3078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.017320 3078 hybrid_clock.cc:648] HybridClock initialized: now 1786533096017296 us; error 33 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.020236 3078 init.cc:377] Logged in from keytab as oryx/127.23.97.195@KRBTEST.COM (short username oryx)
I20260812 11:11:36.021199 3078 webserver.cc:460] Webserver started at http://127.23.97.195:46805/ using document root <none> and password file <none>
I20260812 11:11:36.021716 3078 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.021775 3078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.021945 3078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:36.023936 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "cd34e1af6dde4c7bad2f60a28d7dd132"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "4b3ca900c675deb06bcd894ce42459e2"
server_key_iv: "180f200d1e97fb03ddcdc34ae3484c58"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.024519 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "cd34e1af6dde4c7bad2f60a28d7dd132"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "4b3ca900c675deb06bcd894ce42459e2"
server_key_iv: "180f200d1e97fb03ddcdc34ae3484c58"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.028203 3078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:36.030526 3109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.031688 3078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:36.031787 3078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "cd34e1af6dde4c7bad2f60a28d7dd132"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "4b3ca900c675deb06bcd894ce42459e2"
server_key_iv: "180f200d1e97fb03ddcdc34ae3484c58"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.031875 3078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.045369 3078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.048275 3078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:36.048540 3078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.049166 3078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:36.050009 3078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:36.050057 3078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.050127 3078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:36.050163 3078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.059304 3078 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:38589
I20260812 11:11:36.059360 3320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:38589 every 8 connection(s)
I20260812 11:11:36.060223 3078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:36.067155 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 3078
I20260812 11:11:36.067292 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:36.067616 23943 external_mini_cluster.cc:1428] Setting key 6116832aec5ff49a41e7a366ce0e73c8
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:36.074955 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.062127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54047 (local address 127.23.97.254:45285)
0812 11:11:36.062268 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.062272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.063138 (+ 866us) server_negotiation.cc:407] Connection header received
0812 11:11:36.064619 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.064622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.064685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.064791 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.065965 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.066501 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.068102 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.068393 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.068501 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.071272 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.071275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.071278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.071299 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.072916 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.073446 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.073455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.073456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.073533 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.073893 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.073898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.073899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.074078 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.074192 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.074677 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.074792 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:36.075734 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.062511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:54047)
0812 11:11:36.062983 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.063027 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.064337 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.064978 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.064985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.065340 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.065815 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.065829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.066636 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.066640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.067974 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.067985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.068570 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.068575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.068785 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.069457 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.069477 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.071100 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.073043 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.073049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.073063 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.073328 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.073632 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.073636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.073638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.073783 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.074190 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.074197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.074439 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.074629 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.074954 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260812 11:11:36.076802 3322 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:36.077041 3322 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:36.077533 3322 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:36.078559 2460 ts_manager.cc:194] Registered new tserver with Master: cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589)
I20260812 11:11:36.079126 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:54047
I20260812 11:11:36.081763 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.253.keytab.
Entry for principal oryx/127.23.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.253.keytab.
I20260812 11:11:36.111644 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:43063
--webserver_interface=127.23.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.253.keytab
--principal=oryx/127.23.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:45285,127.23.97.253:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:36.223862 3331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:36.224139 3331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:36.224226 3331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:36.228163 3331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:36.228248 3331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:36.228284 3331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:36.228322 3331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:36.228360 3331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:36.233541 3331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:45285,127.23.97.253:43063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.253.keytab
--principal=oryx/127.23.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
W20260812 11:11:36.233783 3331 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.97.254:45285,127.23.97.253:43063'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 11:11:36.234647 3331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:36.235670 3331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:36.243922 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.243937 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.243933 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.244120 3331 server_base.cc:1034] running on GCE node
I20260812 11:11:36.244853 3331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.245970 3331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.247159 3331 hybrid_clock.cc:648] HybridClock initialized: now 1786533096247143 us; error 33 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.249902 3331 init.cc:377] Logged in from keytab as oryx/127.23.97.253@KRBTEST.COM (short username oryx)
I20260812 11:11:36.250890 3331 webserver.cc:460] Webserver started at http://127.23.97.253:41903/ using document root <none> and password file <none>
I20260812 11:11:36.251448 3331 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.251539 3331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.251775 3331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:36.253819 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/instance:
uuid: "0b51ddac38604dffa6d33d37587227e6"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "d15b6823c02f67c0b117a516d0c6d463"
server_key_iv: "c3505d86b28f283e05a048fbff506b2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.254400 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal/instance:
uuid: "0b51ddac38604dffa6d33d37587227e6"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "d15b6823c02f67c0b117a516d0c6d463"
server_key_iv: "c3505d86b28f283e05a048fbff506b2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.258107 3331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:11:36.260530 3368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.261767 3331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:36.261926 3331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "0b51ddac38604dffa6d33d37587227e6"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "d15b6823c02f67c0b117a516d0c6d463"
server_key_iv: "c3505d86b28f283e05a048fbff506b2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.262053 3331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.281509 3331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.284170 3331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:36.284400 3331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.291347 3331 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.253:43063
I20260812 11:11:36.291635 3461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.253:43063 every 8 connection(s)
I20260812 11:11:36.292330 3331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
I20260812 11:11:36.294996 3462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:36.297694 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 3331
I20260812 11:11:36.299079 3462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.253@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:36.311872 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.301019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.23.97.254:45285)
0812 11:11:36.301139 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.301146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.301548 (+ 402us) server_negotiation.cc:407] Connection header received
0812 11:11:36.301768 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.301772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.301826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.301910 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.302846 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.303603 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.304643 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.304871 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.304982 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.307823 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.307827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.307830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.307857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.309597 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.310181 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.310184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.310185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.310259 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.310637 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.310643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.310647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.310923 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.311010 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.311589 (+ 579us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.311709 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:11:36.312834 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.299171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.23.97.253:43063)
0812 11:11:36.299696 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.299714 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.299765 (+ 51us) server_negotiation.cc:407] Connection header received
0812 11:11:36.300569 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.300594 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.300953 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.301871 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.302270 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.303472 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.304726 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.305260 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.305359 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.308558 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.308564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.308577 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.308601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.310446 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.310928 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.310932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.310933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.311031 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.311361 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.311364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.311366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.311620 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.311809 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.312243 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.312411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":127,"threads_started":1}
I20260812 11:11:36.312834 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.300954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48324)
0812 11:11:36.301419 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.301470 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.301645 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.302069 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.302074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.302493 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.302688 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.302702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.303728 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.303735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.304520 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.304530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.305016 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.305020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.305186 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.305761 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.305783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.307644 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.309728 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.309738 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.309758 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.310080 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.310356 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.310360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.310362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.310541 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.311029 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.311036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.311327 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.311541 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.311854 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":102,"threads_started":1}
I20260812 11:11:36.314677 3462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.253@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:36.322593 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.315115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58798)
0812 11:11:36.315226 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.315239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.315312 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.315527 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.315530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.315684 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.315771 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.315776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.316924 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.316927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.317424 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.317431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.317858 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.317862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.317951 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.318344 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.318355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.319581 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.321382 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.321385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.321386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.321575 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.321842 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.321844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.321846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.321927 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.322241 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.322243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.322315 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.322397 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.322485 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 11:11:36.322786 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.315209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.23.97.253:43063)
0812 11:11:36.315306 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.315310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.315322 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:11:36.315394 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.315398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.315435 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.315516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.315880 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.316808 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.317537 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.317751 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.317858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.319728 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.319732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.319734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.319764 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.321237 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.321674 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.321676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.321677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.321759 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.322007 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.322009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.322010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.322158 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.322299 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.322462 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.322555 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260812 11:11:36.325456 3462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Bootstrap starting.
I20260812 11:11:36.327678 3462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:36.328471 3462 log.cc:824] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:36.330262 3462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: No bootstrap required, opened a new log
I20260812 11:11:36.333289 3462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } }
I20260812 11:11:36.333498 3462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:36.333580 3462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b51ddac38604dffa6d33d37587227e6, State: Initialized, Role: FOLLOWER
I20260812 11:11:36.334017 3462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } }
I20260812 11:11:36.334458 3472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } } }
I20260812 11:11:36.334573 3472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:36.335243 3462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:11:36.339982 3489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:11:36.340058 3489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:11:36.341694 3331 master_runner.cc:428] Detected that this master 0b51ddac38604dffa6d33d37587227e6 is joining an existing cluster
I20260812 11:11:36.341761 3331 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.97.253:43063
I20260812 11:11:36.352470 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.344136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.23.97.254:45285)
0812 11:11:36.344266 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.344270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.344548 (+ 278us) server_negotiation.cc:407] Connection header received
0812 11:11:36.344749 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.344752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.344795 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.344848 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.345515 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.346310 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.347264 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.347499 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.347617 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.348934 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.348938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.348940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.348965 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.350612 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351109 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.351116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.351119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.351215 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351528 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.351532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.351534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.351722 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.351835 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.352154 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.352286 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:11:36.352501 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.343987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58810)
0812 11:11:36.344538 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.344551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.344654 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.344906 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.344909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.345197 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.345409 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.345420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.346421 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.346425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.347371 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.347384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.348047 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.348053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.348159 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.348591 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.348607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.349078 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.351052 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.351060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.351264 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.351556 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.351561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.351635 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.352013 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.352019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.352112 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.352218 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.352317 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":79,"thread_start_us":107,"threads_started":1}
I20260812 11:11:36.352876 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.343986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48334)
0812 11:11:36.344460 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.344472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.344588 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.345007 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.345009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.345228 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.345397 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.345403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.346434 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.346444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.347132 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.347141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.347629 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.347633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.347839 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.348411 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.348433 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.348800 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.350734 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.350738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.350741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.350992 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.351324 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.351334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.351432 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.351824 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.351827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.351929 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.352084 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.352227 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":151,"thread_start_us":114,"threads_started":1}
I20260812 11:11:36.353332 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.344262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.23.97.253:43063)
0812 11:11:36.344405 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.344409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.344629 (+ 220us) server_negotiation.cc:407] Connection header received
0812 11:11:36.344744 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.344747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.344788 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.344865 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.345539 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.346289 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.347497 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.347915 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.348033 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.349232 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.349236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.349238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.349260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.350943 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351373 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.351377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.351379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.351462 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.351719 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.351722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.351723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.351907 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.352010 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.352713 (+ 703us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.352809 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":38}
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.254@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:36.369257 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.357510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.23.97.253:43063)
0812 11:11:36.357630 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.357634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.357945 (+ 311us) server_negotiation.cc:407] Connection header received
0812 11:11:36.358124 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.358127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.358176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.358265 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.358850 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.359878 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.360839 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.361166 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.361337 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.363787 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.363790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.363792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.363812 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.365428 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.365976 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.365981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.365986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.366124 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.368272 (+ 2148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.368277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.368279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.368516 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.368632 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.368968 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.369094 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:36.369275 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.357453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58820)
0812 11:11:36.357810 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.357862 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.357996 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.358304 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.358309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.358641 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.358739 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.358745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.360010 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.360013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.360726 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.360737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.361330 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.361334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.361417 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.361929 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.361941 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.363657 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.365536 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.365545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.365551 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.365864 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.368083 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.368085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.368086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.368168 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.368641 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.368650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.368766 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.368913 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.369116 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20260812 11:11:36.370857 2460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.97.253:43063: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b51ddac38604dffa6d33d37587227e6"
member_type: NON_VOTER
last_known_addr {
host: "127.23.97.253"
port: 43063
}
attrs {
promote: true
}
}
dest_uuid: "59c8040e0cf149b3b173902ea889d681"
cas_config_opid_index: -1
I20260812 11:11:36.371562 2460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } }
I20260812 11:11:36.373536 3514 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } }
I20260812 11:11:36.374109 3420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:36.375273 3514 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.97.253:43063
I20260812 11:11:36.375442 3420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 11:11:36.375581 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } }
I20260812 11:11:36.375669 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:36.376085 3331 master.cc:537] Master@127.23.97.253:43063 shutting down...
I20260812 11:11:36.376724 3515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:11:36.378572 2435 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 11:11:36.379456 2435 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 -> Peer 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253:43063): Couldn't send request to peer 0b51ddac38604dffa6d33d37587227e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:11:36.380697 3331 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:36.381004 3331 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:36.381139 3331 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: stopping tablet replica
WARNING: no policy specified for oryx/127.23.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.252@KRBTEST.COM" created.
I20260812 11:11:36.387019 3331 master.cc:552] Master@127.23.97.253:43063 shutdown complete.
I20260812 11:11:36.387105 3331 master_runner.cc:305] Clearing existing system tablet
I20260812 11:11:36.387382 3331 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.387470 3331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.388578 3331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 11:11:36.389460 3522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.389669 3331 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.389825 3331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:36.389865 3331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "0b51ddac38604dffa6d33d37587227e6"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "d15b6823c02f67c0b117a516d0c6d463"
server_key_iv: "c3505d86b28f283e05a048fbff506b2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.391512 3331 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:36.393836 3331 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:11:36.393951 3331 log.cc:1196] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 11:11:36.394268 3331 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Deleting consensus metadata
I20260812 11:11:36.394455 3331 master_runner.cc:315] Copying system tablet from 127.23.97.254:45285
I20260812 11:11:36.395833 3331 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: tablet copy: Beginning tablet copy session from remote peer at address 127.23.97.254:45285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.252.keytab.
Entry for principal oryx/127.23.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.252.keytab.
I20260812 11:11:36.401645 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:35705
--webserver_interface=127.23.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.252.keytab
--principal=oryx/127.23.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:11:36.404937 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.396888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.23.97.254:45285)
0812 11:11:36.397016 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.397019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.397093 (+ 74us) server_negotiation.cc:407] Connection header received
0812 11:11:36.397243 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.397246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.397290 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.397362 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.397855 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.398429 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.399620 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.399835 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.399914 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.401251 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.401255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.401258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.401285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.403083 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.403620 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.403623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.403624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.403702 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.404016 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.404020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.404021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.404200 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.404319 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.404643 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.404752 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:11:36.405454 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.396706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48350)
0812 11:11:36.396976 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.396992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.397143 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.397404 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.397407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.397618 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.397733 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.397741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.398539 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.398543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.399465 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.399476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.399947 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.399951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.400042 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.400674 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.400690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.401094 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.403215 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.403219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.403221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.403475 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.403811 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.403814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.403817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.403914 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.404308 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.404311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.404452 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.404571 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.404724 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260812 11:11:36.406059 2495 tablet_copy_service.cc:140] P 59c8040e0cf149b3b173902ea889d681: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b51ddac38604dffa6d33d37587227e6 ({username='oryx', principal='oryx/127.23.97.253@KRBTEST.COM'} at 127.0.0.1:48350)
I20260812 11:11:36.406253 2495 tablet_copy_service.cc:161] P 59c8040e0cf149b3b173902ea889d681: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b51ddac38604dffa6d33d37587227e6 at {username='oryx', principal='oryx/127.23.97.253@KRBTEST.COM'} at 127.0.0.1:48350: session id = 0b51ddac38604dffa6d33d37587227e6-00000000000000000000000000000000
I20260812 11:11:36.407789 2495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:11:36.409420 3331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:11:36.411335 3331 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: tablet copy: Starting download of 0 data blocks...
I20260812 11:11:36.411527 3331 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: tablet copy: Starting download of 1 WAL segments...
I20260812 11:11:36.413265 3331 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:11:36.414747 2495 tablet_copy_service.cc:342] P 59c8040e0cf149b3b173902ea889d681: Request end of tablet copy session 0b51ddac38604dffa6d33d37587227e6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.97.253@KRBTEST.COM'} at 127.0.0.1:48350
I20260812 11:11:36.414867 2495 tablet_copy_service.cc:433] P 59c8040e0cf149b3b173902ea889d681: ending tablet copy session 0b51ddac38604dffa6d33d37587227e6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b51ddac38604dffa6d33d37587227e6
I20260812 11:11:36.421783 3331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:36.423404 3552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.423475 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.423666 3331 server_base.cc:1034] running on GCE node
W20260812 11:11:36.424998 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.425259 3331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.425576 3331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.426733 3331 hybrid_clock.cc:648] HybridClock initialized: now 1786533096426744 us; error 43 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.428256 3331 init.cc:377] Logged in from keytab as oryx/127.23.97.253@KRBTEST.COM (short username oryx)
I20260812 11:11:36.428745 3331 webserver.cc:460] Webserver started at http://127.23.97.253:39227/ using document root <none> and password file <none>
I20260812 11:11:36.428954 3331 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.429008 3331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.430071 3331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:36.430980 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.431221 3331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:36.431313 3331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "0b51ddac38604dffa6d33d37587227e6"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "d15b6823c02f67c0b117a516d0c6d463"
server_key_iv: "c3505d86b28f283e05a048fbff506b2a"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.431393 3331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.453231 3331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.455184 3331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.460691 3331 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.253:43063
I20260812 11:11:36.460983 3660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.253:43063 every 8 connection(s)
I20260812 11:11:36.461382 3331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
I20260812 11:11:36.463127 3662 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:11:36.463922 3662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Bootstrap starting.
I20260812 11:11:36.472450 3662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:11:36.472747 3662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: Bootstrap complete.
I20260812 11:11:36.473237 3662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } }
I20260812 11:11:36.473372 3662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b51ddac38604dffa6d33d37587227e6, State: Initialized, Role: LEARNER
I20260812 11:11:36.473491 3662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } }
I20260812 11:11:36.473840 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } }
I20260812 11:11:36.473999 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:11:36.474004 3662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:36.477293 3693 catalog_manager.cc:1259] Loaded cluster ID: 4cf482f93e894d95864228abacc3e8dc
I20260812 11:11:36.477365 3693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: loading cluster ID for follower catalog manager: success
I20260812 11:11:36.478880 3693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: acquiring CA information for follower catalog manager: success
I20260812 11:11:36.479941 3693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:11:36.548125 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:36.548420 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:36.548497 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:36.552429 3540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:36.552479 3540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:36.552497 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:36.552546 3540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:36.552575 3540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:36.557715 3540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.252.keytab
--principal=oryx/127.23.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:36.558722 3540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:36.559712 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:36.568356 3706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.568476 3702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.568356 3708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.568600 3540 server_base.cc:1034] running on GCE node
I20260812 11:11:36.569259 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.570297 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.571460 3540 hybrid_clock.cc:648] HybridClock initialized: now 1786533096571443 us; error 27 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.574296 3540 init.cc:377] Logged in from keytab as oryx/127.23.97.252@KRBTEST.COM (short username oryx)
I20260812 11:11:36.575260 3540 webserver.cc:460] Webserver started at http://127.23.97.252:38591/ using document root <none> and password file <none>
I20260812 11:11:36.575852 3540 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.575919 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.576185 3540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:36.578244 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/instance:
uuid: "a3cf8485be344671842cce9c3e6f17bd"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "93509c09416d1b6c601509039dc25649"
server_key_iv: "0311b08ab2b9b53d823071ce3482684e"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.578774 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal/instance:
uuid: "a3cf8485be344671842cce9c3e6f17bd"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "93509c09416d1b6c601509039dc25649"
server_key_iv: "0311b08ab2b9b53d823071ce3482684e"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.582326 3540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:11:36.584667 3722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.585882 3540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:36.586012 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "a3cf8485be344671842cce9c3e6f17bd"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "93509c09416d1b6c601509039dc25649"
server_key_iv: "0311b08ab2b9b53d823071ce3482684e"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.586130 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.609491 2818 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
I20260812 11:11:36.620131 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.622983 3540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:36.623178 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.630337 3540 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.252:35705
I20260812 11:11:36.630618 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.252:35705 every 8 connection(s)
I20260812 11:11:36.631278 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
I20260812 11:11:36.634109 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:36.637858 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:11:36.639997 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 3540
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.252@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
I20260812 11:11:36.651588 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.639844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.23.97.254:45285)
0812 11:11:36.640007 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.640011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.640791 (+ 780us) server_negotiation.cc:407] Connection header received
0812 11:11:36.641956 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.641959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.642003 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.642071 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.642981 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.643496 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.644651 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.644960 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.645040 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.647546 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.647550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.647554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.647583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.649255 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.649967 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.649970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.649971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.650042 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.650397 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.650400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.650402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.650607 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.650729 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.651288 (+ 559us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.651406 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:11:36.652539 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.641069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.23.97.252:35705)
0812 11:11:36.641417 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.641429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.641469 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:11:36.641737 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.641760 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.642003 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.642190 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.642888 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.643909 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.644861 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.645314 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.645425 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.647994 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.648001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.648019 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.648039 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.650056 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.650508 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.650512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.650514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.650584 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.650877 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.650881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.650882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.651129 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.651337 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.651561 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.651678 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":139,"threads_started":1}
I20260812 11:11:36.652539 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.640149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48358)
0812 11:11:36.640636 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.640686 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.641737 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.642147 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.642152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.642572 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.642811 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.642850 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.643634 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.643638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.644538 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.644548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.645095 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.645099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.645264 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.645868 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.645884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.647382 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.649463 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.649470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.649480 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.649871 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.650134 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.650140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.650145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.650297 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.650738 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.650743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.651029 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.651228 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.651542 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":162,"threads_started":1}
I20260812 11:11:36.653767 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.252@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:36.661478 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.654196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58826)
0812 11:11:36.654300 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.654315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.654385 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.655020 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.655022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.655144 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.655227 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.655232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.656069 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.656071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.656641 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.656649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.657053 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.657055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.657122 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.657597 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.657609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.658746 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.660229 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.660231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.660232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.660481 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.660715 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.660717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.660718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.660796 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.661117 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.661119 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.661187 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.661262 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.661352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 11:11:36.661628 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.654572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.23.97.253:43063)
0812 11:11:36.654820 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.654823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.654835 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:11:36.654872 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.654874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.654913 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.654987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.655333 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.655981 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.656738 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.656961 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.657039 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.658882 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.658885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.658887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.658909 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.660133 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.660561 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.660563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.660565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.660640 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.660853 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.660854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.660856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.660997 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.661129 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.661350 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.661474 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260812 11:11:36.662336 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.252@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
I20260812 11:11:36.669230 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.662762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.0.0.1:34616)
0812 11:11:36.662872 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.662885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.662955 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.663134 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.663136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.663268 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.663333 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.663338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.663979 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.663981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.664541 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.664550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.664953 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.664955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.665008 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.665368 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.665378 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.666439 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.668018 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.668020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.668021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.668194 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.668468 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.668470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.668472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.668544 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.668861 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.668863 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.668931 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.669008 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.669108 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 11:11:36.669308 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.662830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.23.97.252:35705)
0812 11:11:36.662936 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.662939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.662950 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:36.663012 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.663014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.663046 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.663107 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.663425 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.663885 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.664637 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.664863 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.664939 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.666558 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.666561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.666562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.666583 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.667928 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.668275 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.668277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.668278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.668346 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.668623 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.668625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.668626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.668769 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.668852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.669075 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.669158 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:11:36.672276 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Bootstrap starting.
I20260812 11:11:36.674576 3815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:36.675384 3815 log.cc:824] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:36.677309 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: No bootstrap required, opened a new log
I20260812 11:11:36.680171 3815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } }
I20260812 11:11:36.680351 3815 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:36.680451 3815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cf8485be344671842cce9c3e6f17bd, State: Initialized, Role: FOLLOWER
I20260812 11:11:36.680914 3815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } }
I20260812 11:11:36.681357 3825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } } }
I20260812 11:11:36.681495 3825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:36.682091 3815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:11:36.687404 3845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:11:36.687508 3845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:11:36.692806 3540 master_runner.cc:428] Detected that this master a3cf8485be344671842cce9c3e6f17bd is joining an existing cluster
I20260812 11:11:36.692884 3540 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.97.252:35705
I20260812 11:11:36.703838 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.694974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58836)
0812 11:11:36.695409 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.695424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.695521 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.695839 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.695844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.696104 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.696259 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.696268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.697388 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.697391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.698344 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.698354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.699053 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.699187 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.699736 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.699759 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.700249 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.702039 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.702043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.702045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.702394 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.702683 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.702686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.702688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.702795 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.703273 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.703275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.703363 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.703508 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.703646 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":25,"thread_start_us":121,"threads_started":1}
I20260812 11:11:36.703933 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.695277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.23.97.253:43063)
0812 11:11:36.695391 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.695397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.695477 (+ 80us) server_negotiation.cc:407] Connection header received
0812 11:11:36.695616 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.695619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.695671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.695755 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.696434 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.697243 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.698481 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.698817 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.698888 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.700453 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.700460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.700464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.700492 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.701925 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.702485 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.702491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.702495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.702576 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.702916 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.702919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.702921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.703129 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.703199 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.703598 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.703741 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:11:36.706144 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 2587
I20260812 11:11:36.717144 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.694966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.23.97.254:45285)
0812 11:11:36.695183 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.695188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.696036 (+ 848us) server_negotiation.cc:407] Connection header received
0812 11:11:36.696219 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.696222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.696275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.696331 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.697091 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.697916 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699485 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.699810 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699938 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.702411 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.702415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.702418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.702446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.704121 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.704665 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.704668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.704670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.704763 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.705091 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.705094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.705096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.705521 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.705898 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.716747 (+ 10849us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.716924 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260812 11:11:36.717973 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.694979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48374)
0812 11:11:36.695948 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.695961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.696108 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.696527 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.696530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.696809 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.696942 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.696949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.698062 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.698065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.699368 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.699380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.699938 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.700032 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.700809 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.700855 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.702275 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.704234 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.704237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.704240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.704543 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.704867 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.704871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.704873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.704970 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.705941 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.705944 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.706049 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.717556 (+ 11507us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.717718 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":3,"spinlock_wait_cycles":12288,"thread_start_us":107,"threads_started":1}
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533095, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.254@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
I20260812 11:11:36.731997 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.695456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.23.97.252:35705)
0812 11:11:36.695560 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.695564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.696051 (+ 487us) server_negotiation.cc:407] Connection header received
0812 11:11:36.696205 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.696210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.696249 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.696294 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.696939 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.697607 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.698629 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.698930 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699000 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.728187 (+ 29187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.728193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.728199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.728227 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.730016 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.730652 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.730658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.730663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.730743 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.731020 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.731025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.731030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.731240 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.731329 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.731679 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.731802 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:11:36.732863 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.695633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.0.0.1:34628)
0812 11:11:36.695973 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.695988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.696112 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.696433 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.696435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.696708 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.696827 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.696835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.697713 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.697716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.698533 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.698541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.699120 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.699124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.699227 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.727505 (+ 28278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.727526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.728038 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.730142 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.730148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.730153 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.730547 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.730842 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.730848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.730852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.730934 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.731396 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.731401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.731506 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.732407 (+ 901us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.732579 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":5,"spinlock_wait_cycles":6016,"thread_start_us":73,"threads_started":1}
I20260812 11:11:36.735853 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.724346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.0.0.1:34632)
0812 11:11:36.724608 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.724622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.724698 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.725590 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.725593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.725802 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.725905 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.725912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.727055 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.727059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.727932 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.727943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.728595 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.728598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.728695 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.729164 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.729178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.730680 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.733979 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.733982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.733985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.734227 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.734853 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.734857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.734859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.734950 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.735361 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.735365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.735462 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.735541 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.735678 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260812 11:11:36.736045 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:33243
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:45285
--webserver_port=43213
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:11:36.736503 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.724839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.23.97.252:35705)
0812 11:11:36.725286 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.725294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.725307 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:36.725395 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.725398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.725449 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.726017 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.726117 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.726919 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.728086 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.728466 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.728925 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.730909 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.730913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.730915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.730940 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.733853 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.734640 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.734644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.734645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.734729 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.735061 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.735064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.735065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.735248 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.736122 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.736197 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.736309 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":296,"threads_started":1}
I20260812 11:11:36.737320 2460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.97.252:35705: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd"
member_type: NON_VOTER
last_known_addr {
host: "127.23.97.252"
port: 35705
}
attrs {
promote: true
}
}
dest_uuid: "59c8040e0cf149b3b173902ea889d681"
cas_config_opid_index: 5
I20260812 11:11:36.738055 2460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } }
I20260812 11:11:36.739135 3515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3cf8485be344671842cce9c3e6f17bd (127.23.97.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.740715 3515 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.97.252:35705
I20260812 11:11:36.740897 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.740983 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:36.741878 3769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:36.742434 3540 master.cc:537] Master@127.23.97.252:35705 shutting down...
I20260812 11:11:36.743229 3769 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 11:11:36.743634 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.739266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58850)
0812 11:11:36.739441 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.739458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.739533 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.739836 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.739839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.739858 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:36.739936 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.739942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.741069 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.741076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.742256 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.742270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.743190 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.743194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.743303 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.743351 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.743433 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.743501 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 11:11:36.744108 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.739370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.23.97.253:43063)
0812 11:11:36.739488 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.739492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.739506 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:36.739623 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.739626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.739686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.739800 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:36.740054 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.740944 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.742408 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.743054 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.743166 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.743320 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.743897 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.743973 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:11:36.744263 3514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:36.745106 3615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 LEARNER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260812 11:11:36.745927 2434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 -> Peer a3cf8485be344671842cce9c3e6f17bd (127.23.97.252:35705): Couldn't send request to peer a3cf8485be344671842cce9c3e6f17bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:11:36.746428 3514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 11:11:36.748284 3615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3cf8485be344671842cce9c3e6f17bd (127.23.97.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.749257 3825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:36.749413 3825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } }
I20260812 11:11:36.750278 3825 leader_election.cc:288] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285), 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253:43063)
I20260812 11:11:36.750465 3875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.750589 3875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:11:36.750733 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cf8485be344671842cce9c3e6f17bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c8040e0cf149b3b173902ea889d681" is_pre_election: true
I20260812 11:11:36.751201 3615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cf8485be344671842cce9c3e6f17bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b51ddac38604dffa6d33d37587227e6" is_pre_election: true
I20260812 11:11:36.751664 3727 leader_election.cc:302] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cf8485be344671842cce9c3e6f17bd; no voters: 0b51ddac38604dffa6d33d37587227e6, 59c8040e0cf149b3b173902ea889d681
I20260812 11:11:36.751936 3825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:11:36.752014 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c8040e0cf149b3b173902ea889d681. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: NON_VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: true } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.752130 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:11:36.752650 3514 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: attempting to promote NON_VOTER 0b51ddac38604dffa6d33d37587227e6 to VOTER
I20260812 11:11:36.753121 3514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } }
I20260812 11:11:36.754994 3615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 LEARNER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260812 11:11:36.755098 2434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 -> Peer a3cf8485be344671842cce9c3e6f17bd (127.23.97.252:35705): Couldn't send request to peer a3cf8485be344671842cce9c3e6f17bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:11:36.755430 3878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 11:11:36.756011 3540 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:36.756265 3540 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:36.756428 3540 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: stopping tablet replica
I20260812 11:11:36.757179 3515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.758692 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.758806 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:36.759159 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.759265 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:36.759505 3615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.760946 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.761072 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:36.773013 3540 master.cc:552] Master@127.23.97.252:35705 shutdown complete.
I20260812 11:11:36.773079 3540 master_runner.cc:305] Clearing existing system tablet
I20260812 11:11:36.773326 3540 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.773401 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.774539 3540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 11:11:36.775437 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.775641 3540 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.775804 3540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:36.775843 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "a3cf8485be344671842cce9c3e6f17bd"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "93509c09416d1b6c601509039dc25649"
server_key_iv: "0311b08ab2b9b53d823071ce3482684e"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.777683 3540 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:36.779676 3540 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:11:36.779795 3540 log.cc:1196] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 11:11:36.780051 3540 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Deleting consensus metadata
I20260812 11:11:36.780252 3540 master_runner.cc:315] Copying system tablet from 127.23.97.254:45285
I20260812 11:11:36.781440 3540 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: tablet copy: Beginning tablet copy session from remote peer at address 127.23.97.254:45285
I20260812 11:11:36.790035 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.782055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.23.97.254:45285)
0812 11:11:36.782194 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.782198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.782433 (+ 235us) server_negotiation.cc:407] Connection header received
0812 11:11:36.782590 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.782593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.782647 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.782735 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.783304 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.783971 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.784945 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.785185 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.785306 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.786566 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.786570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.786573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.786598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.788281 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.788799 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.788805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.788810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.788901 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.789196 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.789201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.789203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.789393 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.789504 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.789768 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.789875 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:11:36.790048 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.782001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48384)
0812 11:11:36.782330 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.782342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.782479 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.782749 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.782751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.783021 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.783154 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.783165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.784092 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.784096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.784815 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.784825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.785315 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.785318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.785413 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.785944 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.785963 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.786446 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.788415 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.788418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.788420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.788686 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.788994 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.788999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.789001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.789100 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.789494 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.789497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.789601 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.789712 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.789865 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260812 11:11:36.790753 2495 tablet_copy_service.cc:140] P 59c8040e0cf149b3b173902ea889d681: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a3cf8485be344671842cce9c3e6f17bd ({username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:48384)
I20260812 11:11:36.790870 2495 tablet_copy_service.cc:161] P 59c8040e0cf149b3b173902ea889d681: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a3cf8485be344671842cce9c3e6f17bd at {username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:48384: session id = a3cf8485be344671842cce9c3e6f17bd-00000000000000000000000000000000
I20260812 11:11:36.791779 2495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:11:36.793022 3540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:11:36.795413 3540 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: tablet copy: Starting download of 0 data blocks...
I20260812 11:11:36.795600 3540 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: tablet copy: Starting download of 1 WAL segments...
I20260812 11:11:36.797044 3540 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:11:36.798693 2495 tablet_copy_service.cc:342] P 59c8040e0cf149b3b173902ea889d681: Request end of tablet copy session a3cf8485be344671842cce9c3e6f17bd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:48384
I20260812 11:11:36.798786 2495 tablet_copy_service.cc:433] P 59c8040e0cf149b3b173902ea889d681: ending tablet copy session a3cf8485be344671842cce9c3e6f17bd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a3cf8485be344671842cce9c3e6f17bd
I20260812 11:11:36.805605 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:36.808101 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.809352 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.809365 3898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.813264 3540 server_base.cc:1034] running on GCE node
I20260812 11:11:36.813452 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.813772 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.814911 3540 hybrid_clock.cc:648] HybridClock initialized: now 1786533096814898 us; error 45 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.816447 3540 init.cc:377] Logged in from keytab as oryx/127.23.97.252@KRBTEST.COM (short username oryx)
I20260812 11:11:36.816938 3540 webserver.cc:460] Webserver started at http://127.23.97.252:43413/ using document root <none> and password file <none>
I20260812 11:11:36.817139 3540 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.817195 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.818266 3540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:36.819237 3916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.819514 3540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:36.819612 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "a3cf8485be344671842cce9c3e6f17bd"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "93509c09416d1b6c601509039dc25649"
server_key_iv: "0311b08ab2b9b53d823071ce3482684e"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.819702 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.835348 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.837376 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.843041 3540 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.252:35705
I20260812 11:11:36.843060 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.252:35705 every 8 connection(s)
I20260812 11:11:36.843592 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
I20260812 11:11:36.845166 4014 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:11:36.846022 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Bootstrap starting.
I20260812 11:11:36.854031 3070 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
I20260812 11:11:36.854764 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:11:36.855036 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Bootstrap complete.
I20260812 11:11:36.855633 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } }
I20260812 11:11:36.855809 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3cf8485be344671842cce9c3e6f17bd, State: Initialized, Role: LEARNER
I20260812 11:11:36.855942 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } }
I20260812 11:11:36.856271 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:36.856463 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: LEARNER
I20260812 11:11:36.856463 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:36.859521 4040 catalog_manager.cc:1259] Loaded cluster ID: 4cf482f93e894d95864228abacc3e8dc
I20260812 11:11:36.859622 4040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: loading cluster ID for follower catalog manager: success
I20260812 11:11:36.861532 4040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: acquiring CA information for follower catalog manager: success
I20260812 11:11:36.862820 4040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:11:36.911775 3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:36.912009 3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:36.912050 3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:36.915923 3871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:36.915975 3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:36.916031 3871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:36.921387 3871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:33243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=43213
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:36.922279 3871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:36.923204 3871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:11:36.932943 3871 server_base.cc:1034] running on GCE node
W20260812 11:11:36.932907 4046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.932907 4049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:36.932910 4047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:36.933509 3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:36.934114 3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:36.935307 3871 hybrid_clock.cc:648] HybridClock initialized: now 1786533096935297 us; error 31 us; skew 500 ppm
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:36.938135 3871 init.cc:377] Logged in from keytab as oryx/127.23.97.193@KRBTEST.COM (short username oryx)
I20260812 11:11:36.938915 3871 webserver.cc:460] Webserver started at http://127.23.97.193:43213/ using document root <none> and password file <none>
I20260812 11:11:36.939417 3871 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:36.939489 3871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:36.943138 3871 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:36.945088 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.946238 3871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:36.946339 3871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "72202d2e0bb842cf820d756339f6ac31"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "9349f1c6b34a664c33e47c5f603b3b72"
server_key_iv: "7013a1f3cb626fe6ae89d6e84f27dff7"
server_key_version: "encryptionkey@0"
I20260812 11:11:36.946818 3871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:36.960304 3871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:36.963315 3871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:36.963462 3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:36.963959 3871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:36.964897 3871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:36.964957 3871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.965009 3871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:36.965040 3871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:36.973932 3871 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33243
I20260812 11:11:36.973958 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33243 every 8 connection(s)
I20260812 11:11:36.975035 3871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:36.980656 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 3871
I20260812 11:11:36.980873 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 2826
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
Aug 12 11:11:36 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:36.996619 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.976893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:51755 (local address 127.23.97.253:43063)
0812 11:11:36.977025 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.977030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.978452 (+ 1422us) server_negotiation.cc:407] Connection header received
0812 11:11:36.979331 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.979335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.979390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.979442 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.980938 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.981753 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.988153 (+ 6400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.988670 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.988775 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.992106 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.992114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.992120 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.992160 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.994376 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.994890 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.994895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.994899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.994981 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.995309 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.995314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.995318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.995532 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.995597 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.996228 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.996393 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:11:36.997398 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.978699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:46797 (local address 127.23.97.252:35705)
0812 11:11:36.978961 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.978974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.978989 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:36.979287 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.979291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.979350 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.979432 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.980974 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.982066 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.983603 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.983953 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.984046 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.989026 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:36.989032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:36.989037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:36.989081 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:36.992164 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.993263 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.993271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.993275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.994533 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:36.995110 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:36.995117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:36.995122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:36.995460 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:36.995585 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:36.996916 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0812 11:11:36.997098 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260812 11:11:36.998836 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.977373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.23.97.193:51755)
0812 11:11:36.978358 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.978371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.979072 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.979607 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.979613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.980084 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.980757 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.980779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.986748 (+ 5969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.986751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.987962 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.987979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.988984 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.988991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.989156 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.990095 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.990117 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.991899 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.994504 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.994509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.994514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.994762 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.995115 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.995120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.995124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.995208 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.995666 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.995674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.995981 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.996814 (+ 833us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.997850 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":964,"spinlock_wait_cycles":52480,"thread_start_us":195,"threads_started":1}
I20260812 11:11:36.998842 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.978576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.23.97.193:46797)
0812 11:11:36.978871 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.978885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.979087 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.979604 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.979611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.980075 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.980780 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.980795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.982225 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.982230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.983467 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.983479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.984205 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:36.984211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:36.984452 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:36.987007 (+ 2555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:36.987030 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:36.988845 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:36.992300 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.992308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.992326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.992708 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.994674 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:36.994680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:36.994685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:36.994959 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:36.995664 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:36.995675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:36.996003 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 11:11:36.996189 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 11:11:36.997707 (+ 1518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":38,"spinlock_wait_cycles":1408,"thread_start_us":116,"threads_started":1}
I20260812 11:11:37.000607 4283 heartbeater.cc:347] Connected to a master server at 127.23.97.253:43063
I20260812 11:11:37.000926 4283 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.001549 4283 heartbeater.cc:511] Master 127.23.97.253:43063 requested a full tablet report, sending...
I20260812 11:11:37.002146 4279 heartbeater.cc:347] Connected to a master server at 127.23.97.252:35705
I20260812 11:11:37.002269 4279 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.002468 4279 heartbeater.cc:511] Master 127.23.97.252:35705 requested a full tablet report, sending...
I20260812 11:11:37.003185 3593 ts_manager.cc:194] Registered new tserver with Master: 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:37.003669 3942 ts_manager.cc:194] Registered new tserver with Master: 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533096, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:37.009296 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:42505
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:45285
--webserver_port=37091
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:11:37.011314 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.977373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:53591)
0812 11:11:36.977945 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:36.977986 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:36.979072 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:36.979607 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:36.979613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:36.980083 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:36.980785 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:36.980792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.004659 (+ 23867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.004661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.005210 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.005216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.005754 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.005758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.005849 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.006386 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.006398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.007609 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.009739 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.009741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.009743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.009949 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.010259 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.010261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.010262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.010335 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.010786 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.010788 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.010862 (+ 74us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.010979 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.011118 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":101,"spinlock_wait_cycles":35072,"thread_start_us":714,"threads_started":1}
I20260812 11:11:37.011328 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:36.977063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:53591 (local address 127.23.97.254:45285)
0812 11:11:36.977204 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:36.977213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:36.978095 (+ 882us) server_negotiation.cc:407] Connection header received
0812 11:11:36.979318 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:36.979321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:36.979381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:36.979440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:36.980938 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:36.981762 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.005357 (+ 23595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.005605 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.005742 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.007767 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.007771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.007774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.007814 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.009629 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.010052 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.010059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.010064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.010161 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.010424 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.010430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.010435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.010688 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.010794 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.011054 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.011176 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:11:37.012346 4280 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:37.012468 4280 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.012662 4280 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:37.013437 2460 ts_manager.cc:194] Re-registered known tserver with Master: 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:37.014012 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:53591
I20260812 11:11:37.081279 3322 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
W20260812 11:11:37.123921 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:37.124151 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:37.124191 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:37.128119 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:37.128172 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:37.128228 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:37.133509 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:42505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=37091
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:37.134405 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:37.135439 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:11:37.145092 4290 server_base.cc:1034] running on GCE node
W20260812 11:11:37.145058 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:37.145049 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:37.145102 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:37.145797 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:37.146476 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:37.147640 4290 hybrid_clock.cc:648] HybridClock initialized: now 1786533097147629 us; error 26 us; skew 500 ppm
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:37.150774 4290 init.cc:377] Logged in from keytab as oryx/127.23.97.194@KRBTEST.COM (short username oryx)
I20260812 11:11:37.151691 4290 webserver.cc:460] Webserver started at http://127.23.97.194:37091/ using document root <none> and password file <none>
I20260812 11:11:37.152266 4290 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:37.152347 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:37.156111 4290 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:37.158335 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.159585 4290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:11:37.159709 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "d996923ae74d4b2cbc93ce077273bfe1"
format_stamp: "Formatted at 2026-08-12 11:11:35 on dist-test-slave-zpvx"
server_key: "e6b70a909d85a58f745e10be05bd9b2e"
server_key_iv: "5366eb7d428cd81d8d82f3bf02153205"
server_key_version: "encryptionkey@0"
I20260812 11:11:37.160238 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:37.171021 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:37.173660 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:37.173864 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:37.174463 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:37.175375 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:37.175427 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.175488 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:37.175520 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.184556 4290 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:42505
I20260812 11:11:37.184626 4546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:42505 every 8 connection(s)
I20260812 11:11:37.185559 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:37.186597 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 4290
I20260812 11:11:37.186775 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 3078
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
I20260812 11:11:37.212822 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.194222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:52483 (local address 127.23.97.253:43063)
0812 11:11:37.194400 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.194408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.197063 (+ 2655us) server_negotiation.cc:407] Connection header received
0812 11:11:37.198324 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.198330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.198393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.198446 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.200756 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.201604 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.203166 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.203685 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.203866 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.208074 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.208080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.208086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.208117 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.209943 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.210734 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.210740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.210744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.210832 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.211309 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.211314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.211318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.211543 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.211630 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.212469 (+ 839us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.212590 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:11:37.214342 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.187710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.23.97.194:52483)
0812 11:11:37.196911 (+ 9201us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.196965 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.198100 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.198821 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.198831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.199592 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.200528 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.200572 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.201737 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.201743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.203020 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.203034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.203882 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.203888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.204096 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.204893 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.204925 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.207863 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.210080 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.210091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.210112 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.210619 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.210937 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.210944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.210949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.211199 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.211667 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.211678 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.212110 (+ 432us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.212448 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.212922 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8799,"thread_start_us":2086,"threads_started":1}
I20260812 11:11:37.215658 4549 heartbeater.cc:347] Connected to a master server at 127.23.97.253:43063
I20260812 11:11:37.215972 4549 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.216727 4549 heartbeater.cc:511] Master 127.23.97.253:43063 requested a full tablet report, sending...
I20260812 11:11:37.218688 3593 ts_manager.cc:194] Registered new tserver with Master: d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505)
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
I20260812 11:11:37.226956 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:38589
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:45285
--webserver_port=46805
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:11:37.227852 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.217423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:45663 (local address 127.23.97.254:45285)
0812 11:11:37.217587 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.217593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.217710 (+ 117us) server_negotiation.cc:407] Connection header received
0812 11:11:37.217857 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.217863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.217913 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.218051 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.218748 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.219581 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.220826 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.221260 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.221410 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.223610 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.223617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.223622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.223656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.225718 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226265 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.226269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.226271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.226358 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226840 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.226846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.226851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.227093 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.227205 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.227549 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.227684 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:11:37.227895 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.217045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:45663)
0812 11:11:37.217620 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.217641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.217750 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.218186 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.218192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.218383 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.218544 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.218554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.219740 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.219744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.220671 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.220681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.221392 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.221396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.221498 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.222002 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.222022 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.223447 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.225869 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.225873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.225875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.226124 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.226486 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.226491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.226578 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.227257 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.227263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.227370 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.227489 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.227656 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478}
I20260812 11:11:37.228291 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.196706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:40391 (local address 127.23.97.252:35705)
0812 11:11:37.216778 (+ 20072us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.216784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.216801 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:11:37.216845 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.216849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.216901 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.216992 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.220017 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.220847 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.221883 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.222129 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.222241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.224360 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.224363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.224420 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.224448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.226133 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226607 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.226613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.226618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.226710 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.227043 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.227047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.227049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.227285 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.227396 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.227690 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.227821 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19957}
I20260812 11:11:37.228855 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.187710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.23.97.194:40391)
0812 11:11:37.201133 (+ 13423us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.201147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.201231 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.219253 (+ 18022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.219259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.219542 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.219869 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.219883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.220983 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.220988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.221762 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.221772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.222245 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.222248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.222342 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.222840 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.222860 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.224211 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.226246 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.226250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.226496 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.226838 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.226840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.226842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.226921 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.227402 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.227406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.227500 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.228474 (+ 974us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.228648 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13336,"mutex_wait_us":9086,"thread_start_us":1175,"threads_started":1}
I20260812 11:11:37.229249 4548 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:37.229349 4548 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.229532 4547 heartbeater.cc:347] Connected to a master server at 127.23.97.252:35705
I20260812 11:11:37.229560 4548 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:37.229657 4547 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.229866 4547 heartbeater.cc:511] Master 127.23.97.252:35705 requested a full tablet report, sending...
I20260812 11:11:37.230377 2460 ts_manager.cc:194] Re-registered known tserver with Master: d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505)
I20260812 11:11:37.230607 3942 ts_manager.cc:194] Registered new tserver with Master: d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505)
I20260812 11:11:37.231040 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:45663
I20260812 11:11:37.335291 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.331574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.0.0.1:34636)
0812 11:11:37.331865 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.331879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.331979 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.332265 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.332268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.332286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.332454 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.332463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.333348 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.333351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.334130 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.334140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.334840 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.334844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.334939 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.334963 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.335053 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.335139 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260812 11:11:37.335278 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.331665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.23.97.252:35705)
0812 11:11:37.331821 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.331826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.331956 (+ 130us) server_negotiation.cc:407] Connection header received
0812 11:11:37.332062 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.332066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.332132 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.332249 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.332661 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.333215 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.334263 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.334704 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.334830 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.334983 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.335067 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.335140 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:37.335891 3961 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 11:11:37.336242 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c8040e0cf149b3b173902ea889d681. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:37.336405 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: LEARNER
W20260812 11:11:37.344013 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:37.344326 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:37.344458 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:37.348500 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:37.348570 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:37.348671 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:37.353998 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:38589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=46805
--tserver_master_addrs=127.23.97.254:45285,127.23.97.253:43063,127.23.97.252:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:37.355029 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:37.356012 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:37.366282 4561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:37.366307 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:37.366284 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:37.366603 4554 server_base.cc:1034] running on GCE node
I20260812 11:11:37.367062 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:37.367718 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:37.368906 4554 hybrid_clock.cc:648] HybridClock initialized: now 1786533097368886 us; error 32 us; skew 500 ppm
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:37.371534 4554 init.cc:377] Logged in from keytab as oryx/127.23.97.195@KRBTEST.COM (short username oryx)
I20260812 11:11:37.372331 4554 webserver.cc:460] Webserver started at http://127.23.97.195:46805/ using document root <none> and password file <none>
I20260812 11:11:37.372910 4554 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:37.373023 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:37.377029 4554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 11:11:37.378962 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.380188 4554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:37.380343 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "cd34e1af6dde4c7bad2f60a28d7dd132"
format_stamp: "Formatted at 2026-08-12 11:11:36 on dist-test-slave-zpvx"
server_key: "4b3ca900c675deb06bcd894ce42459e2"
server_key_iv: "180f200d1e97fb03ddcdc34ae3484c58"
server_key_version: "encryptionkey@0"
I20260812 11:11:37.380926 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:37.399488 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:37.402319 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:37.402514 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:37.403077 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:37.403903 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:37.403975 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.404045 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:37.404090 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:37.412832 4554 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:38589
I20260812 11:11:37.412856 4810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:38589 every 8 connection(s)
I20260812 11:11:37.413763 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:37.415329 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 4554
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.253@KRBTEST.COM
Aug 12 11:11:37 dist-test-slave-zpvx krb5kdc[2385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533097, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.252@KRBTEST.COM
I20260812 11:11:37.431453 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.418199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.23.97.253:43063)
0812 11:11:37.420125 (+ 1926us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.420128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.420139 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:37.420209 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.420214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.420310 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.420446 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.421606 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422413 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.423535 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.423898 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.423991 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.425981 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.425984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.425987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.426015 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.427879 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.428884 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.428887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.428889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.428975 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.429880 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.429883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.429885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.430114 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.430280 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.431104 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.431237 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1854,"thread_start_us":73,"threads_started":1}
I20260812 11:11:37.434566 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.415564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:35565 (local address 127.23.97.254:45285)
0812 11:11:37.415748 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.415752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.418787 (+ 3035us) server_negotiation.cc:407] Connection header received
0812 11:11:37.419270 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.419275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.419332 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.419383 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.421554 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422305 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.424689 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.425026 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425126 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.428873 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.428879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.428883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.428915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.431299 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.432037 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.432041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.432042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.432416 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.433259 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.433263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.433266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.433500 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.433584 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.434290 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.434405 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 11:11:37.434566 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.417732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57195 (local address 127.23.97.252:35705)
0812 11:11:37.417908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.417914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.418561 (+ 647us) server_negotiation.cc:407] Connection header received
0812 11:11:37.419242 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.419259 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.419317 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.419374 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.422436 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.423539 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425008 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.425338 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425424 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.429697 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.429704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.429710 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.429746 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.431461 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.433081 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.433084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.433086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.433174 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.433472 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.433476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.433478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.433697 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.433776 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.434289 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.434406 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:11:37.436064 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.415743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:41319 (local address 127.23.97.253:43063)
0812 11:11:37.415905 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.415910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.417390 (+ 1480us) server_negotiation.cc:407] Connection header received
0812 11:11:37.419286 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.419289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.419339 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.419393 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.421780 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422491 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.424484 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.424807 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.424891 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.432202 (+ 7311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.432209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.432212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.432241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.433990 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.434644 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.434650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.434653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.434766 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.435087 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.435090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.435092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.435299 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.435382 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.435773 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.435888 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:11:37.436301 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.417357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.23.97.195:57195)
0812 11:11:37.418479 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.418492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.418978 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.419789 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.419798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.421342 (+ 1544us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.421466 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.421476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.423676 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.423679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.424855 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.424871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.425594 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.425741 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.426271 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.426299 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.429515 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.431840 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.431843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.431845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.432324 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.433283 (+ 959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.433285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.433287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.433364 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.433829 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.433831 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.433986 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.434775 (+ 789us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.435248 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":197,"spinlock_wait_cycles":8704,"thread_start_us":90,"threads_started":1}
I20260812 11:11:37.436663 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.416011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:35565)
0812 11:11:37.418701 (+ 2690us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.418717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.418984 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.419788 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.419797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.420309 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.421392 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.421413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422806 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.422811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.424579 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.424585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.425928 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.426028 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.426418 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.426434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.428677 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.431552 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.431560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.431576 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.431918 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.432932 (+ 1014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.432935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.432937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.433136 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.433652 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.433659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.433986 (+ 327us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.434775 (+ 789us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.435199 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":501,"spinlock_wait_cycles":5632,"thread_start_us":933,"threads_started":1}
I20260812 11:11:37.437283 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.416011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.23.97.195:41319)
0812 11:11:37.417213 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.417281 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.418978 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.419789 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.419798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.421554 (+ 1756us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:37.421663 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.421669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422660 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.422663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.424357 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.424392 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.425307 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.425312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.425562 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.426294 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:37.426314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:37.432045 (+ 5731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:37.434262 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.434266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.434268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.434515 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.434879 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:37.434882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:37.434884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:37.434971 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:37.435446 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:37.435451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:37.435551 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.436260 (+ 709us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.436421 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":561,"spinlock_wait_cycles":1792,"thread_start_us":133,"threads_started":1}
I20260812 11:11:37.437520 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.417456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.23.97.254:45285)
0812 11:11:37.417936 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.417940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.418189 (+ 249us) server_negotiation.cc:407] Connection header received
0812 11:11:37.418359 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.418362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.418423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.418530 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.419674 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.420609 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.421862 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.422204 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.422293 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.425205 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.425209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.425212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.425241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.433273 (+ 8032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.433816 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.433820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.433822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.433918 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.435526 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.435529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.435531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.435731 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.435855 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.437251 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.437371 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":78,"threads_started":1}
I20260812 11:11:37.437597 4811 heartbeater.cc:347] Connected to a master server at 127.23.97.252:35705
I20260812 11:11:37.437778 4813 heartbeater.cc:347] Connected to a master server at 127.23.97.254:45285
I20260812 11:11:37.437840 4811 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.437865 4813 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.437868 4814 heartbeater.cc:347] Connected to a master server at 127.23.97.253:43063
I20260812 11:11:37.437947 4814 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:37.438454 4813 heartbeater.cc:511] Master 127.23.97.254:45285 requested a full tablet report, sending...
I20260812 11:11:37.438455 4811 heartbeater.cc:511] Master 127.23.97.252:35705 requested a full tablet report, sending...
I20260812 11:11:37.438454 4814 heartbeater.cc:511] Master 127.23.97.253:43063 requested a full tablet report, sending...
I20260812 11:11:37.439567 2460 ts_manager.cc:194] Re-registered known tserver with Master: cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589)
I20260812 11:11:37.439791 3593 ts_manager.cc:194] Registered new tserver with Master: cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589)
I20260812 11:11:37.439931 3942 ts_manager.cc:194] Registered new tserver with Master: cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589)
I20260812 11:11:37.440279 2460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:35565
I20260812 11:11:37.440330 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.417965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.23.97.252:35705)
0812 11:11:37.420664 (+ 2699us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.420671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.423733 (+ 3062us) server_negotiation.cc:407] Connection header received
0812 11:11:37.423776 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.423779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.423826 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.423908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.427266 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.428050 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.429434 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.429784 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.429880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.431806 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.431811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.431814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.431841 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.435287 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.435776 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.435781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.435783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.435870 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.436693 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.436696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.436698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.439263 (+ 2565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.439389 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.440053 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.440177 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2618,"thread_start_us":83,"threads_started":1}
I20260812 11:11:37.441505 2460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:37.444711 2460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:37.447319 3879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: NON_VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: true } } }
I20260812 11:11:37.447448 3879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:37.450659 3879 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: attempting to promote NON_VOTER a3cf8485be344671842cce9c3e6f17bd to VOTER
I20260812 11:11:37.450986 3879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } }
I20260812 11:11:37.452076 3615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 11:11:37.452425 3961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 LEARNER]: Refusing update from remote peer 59c8040e0cf149b3b173902ea889d681: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 11:11:37.452521 4846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 11:11:37.452745 3879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 11:11:37.454432 4846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a3cf8485be344671842cce9c3e6f17bd (127.23.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.455087 3961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a3cf8485be344671842cce9c3e6f17bd (127.23.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.455847 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.455979 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:37.456132 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.456169 3879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.456225 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:37.456240 3879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:37.456188 3615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a3cf8485be344671842cce9c3e6f17bd (127.23.97.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.457960 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.458122 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:37.462455 4864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59c8040e0cf149b3b173902ea889d681" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:37.462594 4864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:37.465286 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.460036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:38589 (local address 127.0.0.1:43328)
0812 11:11:37.460200 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.460214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.460290 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.461184 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.461187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.461208 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.461326 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.461333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.462724 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.462728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.463916 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.463928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.464838 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.464843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.464945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.464977 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.465069 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.465151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 11:11:37.465312 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.460197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.23.97.195:38589)
0812 11:11:37.460620 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.460649 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.460676 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:11:37.460759 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.460764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.460992 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.461224 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.461433 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.462553 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.464053 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.464673 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.464806 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.464950 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.465102 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.465204 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":125,"threads_started":1}
I20260812 11:11:37.467911 4680 tablet_service.cc:1467] Processing CreateTablet for tablet c8ae18a22eda477689ecd26028d326bb (DEFAULT_TABLE table=test-table [id=681ce95a10884f12a18d34600cdbf97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:37.468869 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ae18a22eda477689ecd26028d326bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:37.476545 4875 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Bootstrap starting.
I20260812 11:11:37.477581 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.460579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42505 (local address 127.0.0.1:44304)
0812 11:11:37.465431 (+ 4852us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.465444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.465513 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.473219 (+ 7706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.473223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.473245 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.473330 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.473336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.475052 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.475056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.476162 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.476175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.477290 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.477294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.477367 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.477386 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.477426 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.477479 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4800,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260812 11:11:37.478153 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.464877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.23.97.193:33243)
0812 11:11:37.466204 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.466221 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.472658 (+ 6437us) server_negotiation.cc:407] Connection header received
0812 11:11:37.472880 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.472886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.473070 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.473187 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.473590 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.474966 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.476297 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.477306 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.477403 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.477551 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.477909 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.478014 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":90,"threads_started":1}
I20260812 11:11:37.478376 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.460768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.23.97.194:42505)
0812 11:11:37.472741 (+ 11973us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.472754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.472778 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:11:37.472853 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.472858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.473048 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.473485 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.473631 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.474909 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.476324 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.477150 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.477879 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.478067 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.478107 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.478171 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11852,"thread_start_us":115,"threads_started":1}
I20260812 11:11:37.478576 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.464700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33243 (local address 127.0.0.1:53678)
0812 11:11:37.472553 (+ 7853us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.472568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.472734 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.473374 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.473377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.473393 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.473475 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.473480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.475142 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.475145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.476114 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.476124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.477657 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.477660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.477749 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.477771 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.478376 (+ 605us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.478456 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7787,"mutex_wait_us":3,"thread_start_us":123,"threads_started":1}
I20260812 11:11:37.478987 4875 tablet_bootstrap.cc:654] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:37.479779 4875 log.cc:824] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:37.481282 4153 tablet_service.cc:1467] Processing CreateTablet for tablet c8ae18a22eda477689ecd26028d326bb (DEFAULT_TABLE table=test-table [id=681ce95a10884f12a18d34600cdbf97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:37.481581 4419 tablet_service.cc:1467] Processing CreateTablet for tablet c8ae18a22eda477689ecd26028d326bb (DEFAULT_TABLE table=test-table [id=681ce95a10884f12a18d34600cdbf97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:37.481999 4875 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: No bootstrap required, opened a new log
I20260812 11:11:37.482200 4875 ts_tablet_manager.cc:1397] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260812 11:11:37.482406 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ae18a22eda477689ecd26028d326bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:37.482410 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ae18a22eda477689ecd26028d326bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:37.485875 4875 raft_consensus.cc:348] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.486107 4875 raft_consensus.cc:374] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:37.486171 4875 raft_consensus.cc:729] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd34e1af6dde4c7bad2f60a28d7dd132, State: Initialized, Role: FOLLOWER
I20260812 11:11:37.486645 4875 consensus_queue.cc:260] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.489174 4875 ts_tablet_manager.cc:1428] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260812 11:11:37.490305 4879 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Bootstrap starting.
I20260812 11:11:37.491264 4878 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Bootstrap starting.
I20260812 11:11:37.491698 4813 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
I20260812 11:11:37.493552 4878 tablet_bootstrap.cc:654] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:37.494472 4878 log.cc:824] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:37.496881 4879 tablet_bootstrap.cc:654] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:37.497694 4879 log.cc:824] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:37.497745 4878 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: No bootstrap required, opened a new log
I20260812 11:11:37.497931 4878 ts_tablet_manager.cc:1397] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 11:11:37.499421 4879 tablet_bootstrap.cc:492] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: No bootstrap required, opened a new log
I20260812 11:11:37.499634 4879 ts_tablet_manager.cc:1397] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260812 11:11:37.501194 4878 raft_consensus.cc:348] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.501410 4878 raft_consensus.cc:374] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:37.501474 4878 raft_consensus.cc:729] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72202d2e0bb842cf820d756339f6ac31, State: Initialized, Role: FOLLOWER
I20260812 11:11:37.501950 4878 consensus_queue.cc:260] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.502554 4280 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
I20260812 11:11:37.502898 4878 ts_tablet_manager.cc:1428] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:37.502740 4879 raft_consensus.cc:348] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.502974 4879 raft_consensus.cc:374] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:37.503026 4879 raft_consensus.cc:729] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d996923ae74d4b2cbc93ce077273bfe1, State: Initialized, Role: FOLLOWER
I20260812 11:11:37.503507 4879 consensus_queue.cc:260] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.504509 4879 ts_tablet_manager.cc:1428] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:11:37.505765 4548 heartbeater.cc:503] Master 127.23.97.254:45285 was elected leader, sending a full tablet report...
I20260812 11:11:37.573907 4887 raft_consensus.cc:482] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:37.574088 4887 raft_consensus.cc:504] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.575400 4887 leader_election.cc:288] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589), 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:37.579457 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.576074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:36723 (local address 127.23.97.193:33243)
0812 11:11:37.576229 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.576234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.576260 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:11:37.576312 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.576314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.576351 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.576460 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.576701 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.577345 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.578492 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.578872 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.579005 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.579057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.579245 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.579325 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:37.579473 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.575626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:38589 (local address 127.23.97.194:43883)
0812 11:11:37.575774 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.575789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.575892 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.576269 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.576271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.576290 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.576436 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.576442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.577237 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.577240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.578316 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.578325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.578989 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.578992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.579098 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.579153 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.579247 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.579324 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":65}
I20260812 11:11:37.579474 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.575720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:43883 (local address 127.23.97.195:38589)
0812 11:11:37.575889 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.575894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.575912 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:11:37.575976 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.575982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.576034 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.576129 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:37.576555 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.577115 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.578448 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.578863 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.578987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.579038 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.579277 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.579343 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:37.579501 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.575938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33243 (local address 127.23.97.194:36723)
0812 11:11:37.576058 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:37.576069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:37.576154 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:37.576474 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:37.576477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:37.576494 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:37.576586 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.576591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.577470 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.577473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.578340 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:37.578349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.579010 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.579014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:37.579108 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.579138 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:11:37.579201 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 11:11:37.579295 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":38}
I20260812 11:11:37.580089 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ae18a22eda477689ecd26028d326bb" candidate_uuid: "d996923ae74d4b2cbc93ce077273bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72202d2e0bb842cf820d756339f6ac31" is_pre_election: true
I20260812 11:11:37.580101 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ae18a22eda477689ecd26028d326bb" candidate_uuid: "d996923ae74d4b2cbc93ce077273bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" is_pre_election: true
I20260812 11:11:37.580418 4184 raft_consensus.cc:2454] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d996923ae74d4b2cbc93ce077273bfe1 in term 0.
I20260812 11:11:37.580418 4721 raft_consensus.cc:2454] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d996923ae74d4b2cbc93ce077273bfe1 in term 0.
I20260812 11:11:37.580899 4318 leader_election.cc:302] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72202d2e0bb842cf820d756339f6ac31, d996923ae74d4b2cbc93ce077273bfe1; no voters:
I20260812 11:11:37.581125 4887 raft_consensus.cc:2781] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:37.581236 4887 raft_consensus.cc:482] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:37.581296 4887 raft_consensus.cc:3037] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:37.582352 4887 raft_consensus.cc:504] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.582676 4887 leader_election.cc:288] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [CANDIDATE]: Term 1 election: Requested vote from peers cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589), 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:37.583092 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ae18a22eda477689ecd26028d326bb" candidate_uuid: "d996923ae74d4b2cbc93ce077273bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72202d2e0bb842cf820d756339f6ac31"
I20260812 11:11:37.583074 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ae18a22eda477689ecd26028d326bb" candidate_uuid: "d996923ae74d4b2cbc93ce077273bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132"
I20260812 11:11:37.583215 4721 raft_consensus.cc:3037] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:37.583214 4184 raft_consensus.cc:3037] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:37.584545 4184 raft_consensus.cc:2454] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d996923ae74d4b2cbc93ce077273bfe1 in term 1.
I20260812 11:11:37.584545 4721 raft_consensus.cc:2454] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d996923ae74d4b2cbc93ce077273bfe1 in term 1.
I20260812 11:11:37.584836 4318 leader_election.cc:302] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72202d2e0bb842cf820d756339f6ac31, d996923ae74d4b2cbc93ce077273bfe1; no voters:
I20260812 11:11:37.585009 4887 raft_consensus.cc:2781] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:37.585216 4887 raft_consensus.cc:686] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 1 LEADER]: Becoming Leader. State: Replica: d996923ae74d4b2cbc93ce077273bfe1, State: Running, Role: LEADER
I20260812 11:11:37.585561 4887 consensus_queue.cc:237] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } }
I20260812 11:11:37.589015 2460 catalog_manager.cc:5495] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "d996923ae74d4b2cbc93ce077273bfe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:37.614691 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.611639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.23.97.194:42505)
0812 11:11:37.611760 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.611763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.611775 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:11:37.611839 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.611841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.611889 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.611972 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:37.612176 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.612717 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.613488 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.613683 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.613760 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.614011 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:37.614400 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:37.614488 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.614551 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.614593 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:11:37.620621 4721 raft_consensus.cc:1261] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Refusing update from remote peer d996923ae74d4b2cbc93ce077273bfe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:37.620760 4184 raft_consensus.cc:1261] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Refusing update from remote peer d996923ae74d4b2cbc93ce077273bfe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:37.621404 4889 consensus_queue.cc:1035] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:37.621588 4887 consensus_queue.cc:1035] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:37.628715 4894 mvcc.cc:205] Tried to move back new op lower bound from 7317639567846957056 to 7317639567712907264. Current Snapshot: MvccSnapshot[applied={T|T < 7317639567846957056}]
I20260812 11:11:37.629580 4898 mvcc.cc:205] Tried to move back new op lower bound from 7317639567846957056 to 7317639567712907264. Current Snapshot: MvccSnapshot[applied={T|T < 7317639567846957056}]
I20260812 11:11:37.635396 4899 mvcc.cc:205] Tried to move back new op lower bound from 7317639567846957056 to 7317639567712907264. Current Snapshot: MvccSnapshot[applied={T|T < 7317639567846957056}]
I20260812 11:11:37.638535 2459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:37.638670 2459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:37.642588 2459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59c8040e0cf149b3b173902ea889d681: Sending DeleteTablet for 3 replicas of tablet c8ae18a22eda477689ecd26028d326bb
I20260812 11:11:37.643297 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ae18a22eda477689ecd26028d326bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:37 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:43328
I20260812 11:11:37.643482 4153 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ae18a22eda477689ecd26028d326bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:37 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:53678
I20260812 11:11:37.643599 4419 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ae18a22eda477689ecd26028d326bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:37 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:44304
I20260812 11:11:37.643960 4918 tablet_replica.cc:330] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: stopping tablet replica
I20260812 11:11:37.644256 4918 raft_consensus.cc:2229] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:37.644577 4918 raft_consensus.cc:2258] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:37.646150 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 2410
I20260812 11:11:37.646211 4918 ts_tablet_manager.cc:1905] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:37.650465 4919 tablet_replica.cc:330] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: stopping tablet replica
I20260812 11:11:37.650801 4919 raft_consensus.cc:2229] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:37.651160 4919 raft_consensus.cc:2258] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:37.652942 4918 ts_tablet_manager.cc:1918] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:37.653079 4918 log.cc:1196] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/c8ae18a22eda477689ecd26028d326bb
I20260812 11:11:37.653517 4918 ts_tablet_manager.cc:1939] T c8ae18a22eda477689ecd26028d326bb P 72202d2e0bb842cf820d756339f6ac31: Deleting consensus metadata
I20260812 11:11:37.657600 4919 ts_tablet_manager.cc:1905] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:37.657918 4916 tablet_replica.cc:330] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: stopping tablet replica
I20260812 11:11:37.658304 4916 raft_consensus.cc:2229] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:37.660732 4916 raft_consensus.cc:2258] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:37.663244 4916 ts_tablet_manager.cc:1905] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:37.666133 4916 ts_tablet_manager.cc:1918] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:37.666242 4916 log.cc:1196] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/c8ae18a22eda477689ecd26028d326bb
I20260812 11:11:37.666603 4916 ts_tablet_manager.cc:1939] T c8ae18a22eda477689ecd26028d326bb P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting consensus metadata
I20260812 11:11:37.668697 4919 ts_tablet_manager.cc:1918] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:37.668812 4919 log.cc:1196] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/c8ae18a22eda477689ecd26028d326bb
I20260812 11:11:37.669174 4919 ts_tablet_manager.cc:1939] T c8ae18a22eda477689ecd26028d326bb P d996923ae74d4b2cbc93ce077273bfe1: Deleting consensus metadata
W20260812 11:11:37.671343 4324 connection.cc:535] server connection from 127.0.0.1:44304 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:11:37.671353 4071 connection.cc:535] server connection from 127.0.0.1:53678 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 11:11:37.671757 4588 connection.cc:535] server connection from 127.0.0.1:43328 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:11:37.671977 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.644607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.23.97.253:43063)
0812 11:11:37.644719 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.644724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.660711 (+ 15987us) server_negotiation.cc:407] Connection header received
0812 11:11:37.660791 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.660793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.660878 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.660973 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.662202 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.663057 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.664576 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.665024 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.665167 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.667636 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.667643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.667649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.667682 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.669690 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.670263 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.670269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.670274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.670364 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.670681 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.670687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.670691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.670981 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.671119 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.671593 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.671736 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:11:37.683032 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.645247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.23.97.252:35705)
0812 11:11:37.645363 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.645367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.672981 (+ 27614us) server_negotiation.cc:407] Connection header received
0812 11:11:37.673177 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.673180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.673234 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.673289 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.674303 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.675071 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.676262 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.676679 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.676772 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.678483 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.678487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.678490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.678515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.680206 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.681299 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.681304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.681308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.681401 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.681740 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.681744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.681746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.681971 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.682086 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.682661 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.682827 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260812 11:11:37.685959 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.675520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.23.97.252:35705)
0812 11:11:37.675630 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.675634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.676490 (+ 856us) server_negotiation.cc:407] Connection header received
0812 11:11:37.676643 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.676646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.676688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.676748 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.677717 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.678503 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.679832 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.680195 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.680282 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.682003 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.682010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.682014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.682038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.684070 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.684622 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.684627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.684630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.684719 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.685086 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.685089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.685091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.685289 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.685425 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.685666 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.685779 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":43}
I20260812 11:11:37.686118 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:37.675094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.23.97.253:43063)
0812 11:11:37.675522 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:37.675526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:37.676235 (+ 709us) server_negotiation.cc:407] Connection header received
0812 11:11:37.676456 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:37.676459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:37.676501 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:37.676548 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:37.677484 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.678229 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.679429 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:37.679763 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:37.679834 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:37.681515 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:37.681519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:37.681521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:37.681546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:37.683431 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.684749 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.684752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.684754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.684839 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:37.685141 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:37.685144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:37.685146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:37.685371 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:37.685479 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:37.685851 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 11:11:37.685963 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260812 11:11:38.633090 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:38.632524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:50165)
0812 11:11:38.632852 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:38.632970 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
W20260812 11:11:38.633514 4280 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:45285 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:38.636595 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:38.636099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:51475)
0812 11:11:38.636399 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:38.636495 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
W20260812 11:11:38.637044 4813 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:45285 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:38.637184 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:38.636321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:57289)
0812 11:11:38.636846 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:38.636937 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":82,"threads_started":1}
W20260812 11:11:38.637471 4548 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:45285 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.142623 4944 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59c8040e0cf149b3b173902ea889d681)
I20260812 11:11:39.142759 4944 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } }
I20260812 11:11:39.143203 4945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59c8040e0cf149b3b173902ea889d681)
I20260812 11:11:39.143389 4944 leader_election.cc:288] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285), 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253:43063)
I20260812 11:11:39.143425 4945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } }
I20260812 11:11:39.144019 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.143493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48808)
0812 11:11:39.143813 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.143905 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
W20260812 11:11:39.144217 3922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.144717 4945 leader_election.cc:288] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285), a3cf8485be344671842cce9c3e6f17bd (127.23.97.252:35705)
I20260812 11:11:39.145182 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.144801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48816)
0812 11:11:39.145020 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.145093 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
W20260812 11:11:39.145439 3566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.146193 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.145983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48830)
0812 11:11:39.146075 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.146127 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 11:11:39.146389 3922 leader_election.cc:334] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285): Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.146453 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.146197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48834)
0812 11:11:39.146331 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.146382 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 11:11:39.146541 3566 leader_election.cc:334] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285): Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.148258 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.143741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43063 (local address 127.0.0.1:58888)
0812 11:11:39.143966 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.143979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.144080 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.144621 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.144627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.144649 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.144755 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.144759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.145940 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.145944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.147158 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.147169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.147831 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.147835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.147923 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.147948 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.148032 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.148114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260812 11:11:39.148265 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.143944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.23.97.253:43063)
0812 11:11:39.144327 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.144332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.144347 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:39.144421 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.144424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.144485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.144887 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.145019 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.145808 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.147281 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.147717 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.147797 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.147845 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.148053 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.148109 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":303,"thread_start_us":202,"threads_started":1}
I20260812 11:11:39.148711 3615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cf8485be344671842cce9c3e6f17bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b51ddac38604dffa6d33d37587227e6" is_pre_election: true
I20260812 11:11:39.148886 3615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cf8485be344671842cce9c3e6f17bd in term 1.
I20260812 11:11:39.149259 3924 leader_election.cc:302] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b51ddac38604dffa6d33d37587227e6, a3cf8485be344671842cce9c3e6f17bd; no voters: 59c8040e0cf149b3b173902ea889d681
I20260812 11:11:39.149386 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.145138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:35705 (local address 127.0.0.1:34658)
0812 11:11:39.145415 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.145428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.145507 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.145884 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.145889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.145913 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.146006 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.146013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.147018 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.147021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.148057 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.148067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.148925 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.148928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.149040 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.149081 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.149164 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.149241 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":21,"thread_start_us":66,"threads_started":1}
I20260812 11:11:39.149410 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.145297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.23.97.252:35705)
0812 11:11:39.145618 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.145630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.145647 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:11:39.145695 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.145699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.145745 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.146118 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.146232 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.146907 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.148182 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.148774 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.148927 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.148986 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.149209 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.149304 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260812 11:11:39.149495 4944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 11:11:39.149557 4944 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59c8040e0cf149b3b173902ea889d681)
I20260812 11:11:39.149619 4944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:11:39.150051 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b51ddac38604dffa6d33d37587227e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a3cf8485be344671842cce9c3e6f17bd" is_pre_election: true
I20260812 11:11:39.150872 4944 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } }
I20260812 11:11:39.151113 3961 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b51ddac38604dffa6d33d37587227e6 in current term 2: Already voted for candidate a3cf8485be344671842cce9c3e6f17bd in this term.
I20260812 11:11:39.151252 4944 leader_election.cc:288] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 election: Requested vote from peers 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285), 0b51ddac38604dffa6d33d37587227e6 (127.23.97.253:43063)
I20260812 11:11:39.151556 3566 leader_election.cc:302] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b51ddac38604dffa6d33d37587227e6; no voters: 59c8040e0cf149b3b173902ea889d681, a3cf8485be344671842cce9c3e6f17bd
I20260812 11:11:39.151670 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.151464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48842)
0812 11:11:39.151554 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.151599 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 11:11:39.151798 4945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:11:39.152098 3615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3cf8485be344671842cce9c3e6f17bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b51ddac38604dffa6d33d37587227e6"
I20260812 11:11:39.152345 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.152125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48858)
0812 11:11:39.152244 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.152292 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 11:11:39.152459 3922 leader_election.cc:334] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285): Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
I20260812 11:11:39.152967 4945 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:11:39.153951 3615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cf8485be344671842cce9c3e6f17bd in term 2.
I20260812 11:11:39.154227 3924 leader_election.cc:302] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b51ddac38604dffa6d33d37587227e6, a3cf8485be344671842cce9c3e6f17bd; no voters: 59c8040e0cf149b3b173902ea889d681
I20260812 11:11:39.154404 4944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:11:39.154608 4944 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [term 2 LEADER]: Becoming Leader. State: Replica: a3cf8485be344671842cce9c3e6f17bd, State: Running, Role: LEADER
I20260812 11:11:39.154867 4944 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } }
I20260812 11:11:39.156191 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cf8485be344671842cce9c3e6f17bd. Latest consensus state: current_term: 2 leader_uuid: "a3cf8485be344671842cce9c3e6f17bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:39.156328 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:39.156644 4969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:39.159137 4969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:39.159530 4969 catalog_manager.cc:1259] Loaded cluster ID: 4cf482f93e894d95864228abacc3e8dc
I20260812 11:11:39.159600 4969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:39.160117 4969 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:39.160569 4969 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Loaded TSK: 0
I20260812 11:11:39.161201 4969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:11:39.195127 3942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:39.198256 3942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:39.201059 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.200826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48872)
0812 11:11:39.200943 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.201000 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 11:11:39.201225 3615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [term 2 FOLLOWER]: Refusing update from remote peer a3cf8485be344671842cce9c3e6f17bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 11:11:39.201848 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.201540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.0.0.1:48888)
0812 11:11:39.201668 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.201751 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 11:11:39.201766 4944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 11:11:39.201987 3922 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd -> Peer 59c8040e0cf149b3b173902ea889d681 (127.23.97.254:45285): Couldn't send request to peer 59c8040e0cf149b3b173902ea889d681. Status: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:11:39.203275 4945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cf8485be344671842cce9c3e6f17bd. Latest consensus state: current_term: 2 leader_uuid: "a3cf8485be344671842cce9c3e6f17bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:39.203418 4945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:39.204571 4944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a3cf8485be344671842cce9c3e6f17bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:39.204710 4944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:39.204900 4945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a3cf8485be344671842cce9c3e6f17bd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59c8040e0cf149b3b173902ea889d681" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 45285 } } peers { permanent_uuid: "0b51ddac38604dffa6d33d37587227e6" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43063 } attrs { promote: false } } peers { permanent_uuid: "a3cf8485be344671842cce9c3e6f17bd" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 35705 } attrs { promote: false } } }
I20260812 11:11:39.205020 4945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b51ddac38604dffa6d33d37587227e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:11:39.205719 4973 mvcc.cc:205] Tried to move back new op lower bound from 7317639574323335168 to 7317639574140698624. Current Snapshot: MvccSnapshot[applied={T|T < 7317639574323335168 or (T in {7317639574323335168})}]
I20260812 11:11:39.214851 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.210555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42505 (local address 127.0.0.1:44318)
0812 11:11:39.210670 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.210685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.210760 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.211209 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.211214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.211236 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.211356 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.211363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.212487 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.212490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.213751 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.213766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.214411 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.214414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.214534 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.214571 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.214659 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.214731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":82}
I20260812 11:11:39.214890 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.210684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.23.97.194:42505)
0812 11:11:39.210954 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.210957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.210968 (+ 11us) server_negotiation.cc:407] Connection header received
0812 11:11:39.211012 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.211014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.211073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.211473 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.211594 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.212334 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.213873 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.214291 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.214412 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.214479 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.214676 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.214755 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260812 11:11:39.215504 4419 tablet_service.cc:1467] Processing CreateTablet for tablet f6b654aaa4564c1a8c00482f36e07887 (DEFAULT_TABLE table=test-table [id=c572bef797ae481eb6b3b7776dcf231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:39.215817 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b654aaa4564c1a8c00482f36e07887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:39.216250 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.211330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:38589 (local address 127.0.0.1:43336)
0812 11:11:39.211713 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.211724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.211837 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.212465 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.212469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.212486 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.212649 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.212658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.213396 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.213403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.214571 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.214581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.215833 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.215839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.215924 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.215947 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.216018 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.216084 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260812 11:11:39.216420 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.211100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33243 (local address 127.0.0.1:53694)
0812 11:11:39.211215 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.211231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.211311 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.212224 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.212227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.212245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.212330 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.212336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.213481 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.213484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.214718 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.214726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.215989 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.215993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.216099 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.216116 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.216207 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.216280 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":66}
I20260812 11:11:39.216459 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.211482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.23.97.193:33243)
0812 11:11:39.211854 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.211858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.211872 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:39.211922 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.211924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.211968 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.212468 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.212594 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.213349 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.215299 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.215816 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.215947 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.216006 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.216227 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.216310 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":76,"threads_started":1}
I20260812 11:11:39.216745 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.211475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.23.97.195:38589)
0812 11:11:39.211713 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.211717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.211809 (+ 92us) server_negotiation.cc:407] Connection header received
0812 11:11:39.211963 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.211968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.212020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.212130 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.212756 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.213286 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.214723 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.215568 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.215686 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.215769 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.216530 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.216625 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260812 11:11:39.217257 4680 tablet_service.cc:1467] Processing CreateTablet for tablet f6b654aaa4564c1a8c00482f36e07887 (DEFAULT_TABLE table=test-table [id=c572bef797ae481eb6b3b7776dcf231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:39.217535 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b654aaa4564c1a8c00482f36e07887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:39.218494 4988 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Bootstrap starting.
I20260812 11:11:39.218645 4153 tablet_service.cc:1467] Processing CreateTablet for tablet f6b654aaa4564c1a8c00482f36e07887 (DEFAULT_TABLE table=test-table [id=c572bef797ae481eb6b3b7776dcf231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:39.218909 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b654aaa4564c1a8c00482f36e07887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:39.219468 4988 tablet_bootstrap.cc:654] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:39.220098 4989 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Bootstrap starting.
I20260812 11:11:39.221123 4989 tablet_bootstrap.cc:654] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:39.221150 4988 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: No bootstrap required, opened a new log
I20260812 11:11:39.221239 4988 ts_tablet_manager.cc:1397] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:39.221590 4992 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: Bootstrap starting.
I20260812 11:11:39.221717 4988 raft_consensus.cc:348] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.221850 4988 raft_consensus.cc:374] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:39.221894 4988 raft_consensus.cc:729] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d996923ae74d4b2cbc93ce077273bfe1, State: Initialized, Role: FOLLOWER
I20260812 11:11:39.222011 4988 consensus_queue.cc:260] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.222345 4989 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: No bootstrap required, opened a new log
I20260812 11:11:39.222366 4988 ts_tablet_manager.cc:1428] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:39.222434 4989 ts_tablet_manager.cc:1397] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:39.222620 4992 tablet_bootstrap.cc:654] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:39.222925 4989 raft_consensus.cc:348] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.223044 4989 raft_consensus.cc:374] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:39.223085 4989 raft_consensus.cc:729] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd34e1af6dde4c7bad2f60a28d7dd132, State: Initialized, Role: FOLLOWER
I20260812 11:11:39.223210 4989 consensus_queue.cc:260] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.223523 4989 ts_tablet_manager.cc:1428] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:39.223559 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.223017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:39753)
0812 11:11:39.223394 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.223454 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260812 11:11:39.224248 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.224000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:44033)
0812 11:11:39.224132 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.224184 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 11:11:39.225683 4547 heartbeater.cc:503] Master 127.23.97.252:35705 was elected leader, sending a full tablet report...
I20260812 11:11:39.225829 4992 tablet_bootstrap.cc:492] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: No bootstrap required, opened a new log
I20260812 11:11:39.225906 4992 ts_tablet_manager.cc:1397] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 11:11:39.226094 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.224599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:55929)
0812 11:11:39.225936 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.225998 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1273,"thread_start_us":93,"threads_started":1}
I20260812 11:11:39.226356 4992 raft_consensus.cc:348] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.226447 4992 raft_consensus.cc:374] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:39.226470 4992 raft_consensus.cc:729] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72202d2e0bb842cf820d756339f6ac31, State: Initialized, Role: FOLLOWER
I20260812 11:11:39.226596 4992 consensus_queue.cc:260] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.226912 4992 ts_tablet_manager.cc:1428] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:39.227708 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.227332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:52015)
0812 11:11:39.227553 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.227613 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260812 11:11:39.227829 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.227558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:60051)
0812 11:11:39.227715 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.227765 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260812 11:11:39.228340 4284 tablet.cc:2369] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:39.228645 4279 heartbeater.cc:503] Master 127.23.97.252:35705 was elected leader, sending a full tablet report...
I20260812 11:11:39.228785 4811 heartbeater.cc:503] Master 127.23.97.252:35705 was elected leader, sending a full tablet report...
I20260812 11:11:39.230252 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.228272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:52985)
0812 11:11:39.230114 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.230166 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1753,"thread_start_us":118,"threads_started":1}
I20260812 11:11:39.345844 5005 raft_consensus.cc:482] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:39.345996 5005 raft_consensus.cc:504] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.347210 5005 leader_election.cc:288] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505), 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:39.351313 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.347602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:38299 (local address 127.23.97.193:33243)
0812 11:11:39.347735 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.347740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.347969 (+ 229us) server_negotiation.cc:407] Connection header received
0812 11:11:39.348136 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.348144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.348187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.348276 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.348574 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.349086 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350204 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.350586 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.350780 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.351078 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.351168 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:11:39.351313 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.347561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57977 (local address 127.23.97.194:42505)
0812 11:11:39.347769 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.347774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.347790 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:11:39.347952 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.347954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.347998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.348088 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:39.348362 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.348929 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350172 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.350564 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350705 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.350771 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.351098 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.351180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 11:11:39.351315 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.347453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42505 (local address 127.23.97.195:57977)
0812 11:11:39.347688 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.347705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.347814 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.348104 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.348106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.348126 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.348241 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.348246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.349056 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.349072 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.350036 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.350046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.350720 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.350887 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.350939 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.351033 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.351176 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":81}
I20260812 11:11:39.351336 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.347489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33243 (local address 127.23.97.195:38299)
0812 11:11:39.347872 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.347884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:39.348010 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:39.348290 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:39.348293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:39.348310 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:39.348454 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.348461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.349202 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.349207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.350046 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:39.350056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.350719 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.350724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:39.350924 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.350948 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:11:39.351049 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:11:39.351181 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":85,"thread_start_us":162,"threads_started":1}
I20260812 11:11:39.351824 4456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b654aaa4564c1a8c00482f36e07887" candidate_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d996923ae74d4b2cbc93ce077273bfe1" is_pre_election: true
I20260812 11:11:39.352025 4456 raft_consensus.cc:2454] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd34e1af6dde4c7bad2f60a28d7dd132 in term 0.
I20260812 11:11:39.352182 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b654aaa4564c1a8c00482f36e07887" candidate_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72202d2e0bb842cf820d756339f6ac31" is_pre_election: true
I20260812 11:11:39.352367 4184 raft_consensus.cc:2454] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd34e1af6dde4c7bad2f60a28d7dd132 in term 0.
I20260812 11:11:39.352506 4589 leader_election.cc:302] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd34e1af6dde4c7bad2f60a28d7dd132, d996923ae74d4b2cbc93ce077273bfe1; no voters:
I20260812 11:11:39.352772 5005 raft_consensus.cc:2781] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:39.352850 5005 raft_consensus.cc:482] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:39.352907 5005 raft_consensus.cc:3037] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:39.353937 5005 raft_consensus.cc:504] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.354343 5005 leader_election.cc:288] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [CANDIDATE]: Term 1 election: Requested vote from peers d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505), 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243)
I20260812 11:11:39.354617 4456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b654aaa4564c1a8c00482f36e07887" candidate_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d996923ae74d4b2cbc93ce077273bfe1"
I20260812 11:11:39.354651 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b654aaa4564c1a8c00482f36e07887" candidate_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72202d2e0bb842cf820d756339f6ac31"
I20260812 11:11:39.354730 4456 raft_consensus.cc:3037] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:39.354775 4184 raft_consensus.cc:3037] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:39.355731 4184 raft_consensus.cc:2454] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd34e1af6dde4c7bad2f60a28d7dd132 in term 1.
I20260812 11:11:39.355805 4456 raft_consensus.cc:2454] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd34e1af6dde4c7bad2f60a28d7dd132 in term 1.
I20260812 11:11:39.356078 4584 leader_election.cc:302] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72202d2e0bb842cf820d756339f6ac31, cd34e1af6dde4c7bad2f60a28d7dd132; no voters:
I20260812 11:11:39.356305 5005 raft_consensus.cc:2781] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:39.356561 5005 raft_consensus.cc:686] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 LEADER]: Becoming Leader. State: Replica: cd34e1af6dde4c7bad2f60a28d7dd132, State: Running, Role: LEADER
I20260812 11:11:39.356838 5005 consensus_queue.cc:237] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } }
I20260812 11:11:39.357295 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.357042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:34987)
0812 11:11:39.357153 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.357208 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 11:11:39.359505 3942 catalog_manager.cc:5495] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd34e1af6dde4c7bad2f60a28d7dd132" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 38589 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:39.373201 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.369827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.23.97.195:38589)
0812 11:11:39.369949 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:39.369953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:39.369966 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:39.370109 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:39.370111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:39.370169 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:39.370255 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:39.370511 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.370992 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.371821 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:39.372025 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:39.372103 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:39.372436 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:39.372849 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:39.372979 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:39.373037 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 11:11:39.373083 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:11:39.376108 4184 raft_consensus.cc:1261] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Refusing update from remote peer cd34e1af6dde4c7bad2f60a28d7dd132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:39.376338 4456 raft_consensus.cc:1261] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Refusing update from remote peer cd34e1af6dde4c7bad2f60a28d7dd132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:39.376694 5020 consensus_queue.cc:1035] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72202d2e0bb842cf820d756339f6ac31" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:39.376888 5005 consensus_queue.cc:1035] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d996923ae74d4b2cbc93ce077273bfe1" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:39.378925 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.378561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:32905)
0812 11:11:39.378782 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.378846 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260812 11:11:39.379834 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.379297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:57583)
0812 11:11:39.379706 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.379764 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260812 11:11:39.383132 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.382869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.194:43891)
0812 11:11:39.383012 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.383063 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 11:11:39.383265 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.382845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.193:58627)
0812 11:11:39.383151 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.383195 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260812 11:11:39.383276 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:39.381815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:45285 (local address 127.23.97.195:53933)
0812 11:11:39.383065 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.383125 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1186}
I20260812 11:11:39.390858 3942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:39.390993 3942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:39.394701 3942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3cf8485be344671842cce9c3e6f17bd: Sending DeleteTablet for 3 replicas of tablet f6b654aaa4564c1a8c00482f36e07887
I20260812 11:11:39.395349 4419 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b654aaa4564c1a8c00482f36e07887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:39 UTC) from {username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:44318
I20260812 11:11:39.395550 4153 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b654aaa4564c1a8c00482f36e07887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:39 UTC) from {username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:53694
I20260812 11:11:39.395638 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b654aaa4564c1a8c00482f36e07887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:39 UTC) from {username='oryx', principal='oryx/127.23.97.252@KRBTEST.COM'} at 127.0.0.1:43336
I20260812 11:11:39.395889 5049 tablet_replica.cc:330] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31: stopping tablet replica
I20260812 11:11:39.396015 5049 raft_consensus.cc:2229] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:39.396054 5046 tablet_replica.cc:330] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: stopping tablet replica
I20260812 11:11:39.396139 5049 raft_consensus.cc:2258] T f6b654aaa4564c1a8c00482f36e07887 P 72202d2e0bb842cf820d756339f6ac31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:39.396155 5046 raft_consensus.cc:2229] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:39.396265 5046 raft_consensus.cc:2258] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:39.396500 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 3871
I20260812 11:11:39.397266 5050 tablet_replica.cc:330] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: stopping tablet replica
I20260812 11:11:39.397383 5050 raft_consensus.cc:2229] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:39.397614 5050 raft_consensus.cc:2258] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:39.400955 5046 ts_tablet_manager.cc:1905] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:39.402992 5046 ts_tablet_manager.cc:1918] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:39.403047 5046 log.cc:1196] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/f6b654aaa4564c1a8c00482f36e07887
I20260812 11:11:39.403266 5046 ts_tablet_manager.cc:1939] T f6b654aaa4564c1a8c00482f36e07887 P d996923ae74d4b2cbc93ce077273bfe1: Deleting consensus metadata
I20260812 11:11:39.403986 3931 catalog_manager.cc:4841] TS d996923ae74d4b2cbc93ce077273bfe1 (127.23.97.194:42505): tablet f6b654aaa4564c1a8c00482f36e07887 (table test-table [id=c572bef797ae481eb6b3b7776dcf231d]) successfully deleted
I20260812 11:11:39.404196 5050 ts_tablet_manager.cc:1905] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:39.406286 5050 ts_tablet_manager.cc:1918] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:39.406381 5050 log.cc:1196] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/f6b654aaa4564c1a8c00482f36e07887
I20260812 11:11:39.406639 5050 ts_tablet_manager.cc:1939] T f6b654aaa4564c1a8c00482f36e07887 P cd34e1af6dde4c7bad2f60a28d7dd132: Deleting consensus metadata
I20260812 11:11:39.407296 3921 catalog_manager.cc:4841] TS cd34e1af6dde4c7bad2f60a28d7dd132 (127.23.97.195:38589): tablet f6b654aaa4564c1a8c00482f36e07887 (table test-table [id=c572bef797ae481eb6b3b7776dcf231d]) successfully deleted
W20260812 11:11:39.413439 4946 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:39.413095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33243 (local address 127.0.0.1:53710)
0812 11:11:39.413285 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:39.413337 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:33243: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 11:11:39.413658 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 4290
W20260812 11:11:39.413715 3921 catalog_manager.cc:4584] TS 72202d2e0bb842cf820d756339f6ac31 (127.23.97.193:33243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6b654aaa4564c1a8c00482f36e07887: Network error: Client connection negotiation failed: client connection to 127.23.97.193:33243: connect: Connection reset by peer (error 104)
I20260812 11:11:39.436245 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 4554
I20260812 11:11:39.450361 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 3331
I20260812 11:11:39.463271 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 3540
2026-08-12T11:11:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5800 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:39 dist-test-slave-zpvx krb5kdc[5064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:39 dist-test-slave-zpvx krb5kdc[5064](info): set up 2 sockets
Aug 12 11:11:39 dist-test-slave-zpvx krb5kdc[5064](info): commencing operation
krb5kdc: starting...
W20260812 11:11:40.784138 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:40 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab.
Entry for principal oryx/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab.
I20260812 11:11:40.910607 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36971
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab
--principal=oryx/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:41.021119 5085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:41.021366 5085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:41.021405 5085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:41.025213 5085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:41.025259 5085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:11:41.025276 5085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:41.025295 5085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:41.025310 5085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:41.025369 5085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:41.030437 5085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:36971
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.254.keytab
--principal=oryx/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:41.031404 5085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:41.032426 5085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:41.040792 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.040789 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.040784 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.041117 5085 server_base.cc:1034] running on GCE node
I20260812 11:11:41.041769 5085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:41.042819 5085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:41.044011 5085 hybrid_clock.cc:648] HybridClock initialized: now 1786533101043994 us; error 30 us; skew 500 ppm
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:41.050846 5085 init.cc:377] Logged in from keytab as oryx/127.23.97.254@KRBTEST.COM (short username oryx)
I20260812 11:11:41.051754 5085 webserver.cc:460] Webserver started at http://127.23.97.254:45349/ using document root <none> and password file <none>
I20260812 11:11:41.052320 5085 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:41.052438 5085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:41.052676 5085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:41.054634 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "0278565299a245369695a50050293d15"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "c7b6fea2a766918d1f2ee85bdd1fb892"
server_key_iv: "7631be91762555611b594b29ca603e01"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.055195 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "0278565299a245369695a50050293d15"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "c7b6fea2a766918d1f2ee85bdd1fb892"
server_key_iv: "7631be91762555611b594b29ca603e01"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.058902 5085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260812 11:11:41.061368 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.062604 5085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:11:41.062781 5085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "0278565299a245369695a50050293d15"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "c7b6fea2a766918d1f2ee85bdd1fb892"
server_key_iv: "7631be91762555611b594b29ca603e01"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.062904 5085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:41.092718 5085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:41.095541 5085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:41.095757 5085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:41.104220 5085 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:36971
I20260812 11:11:41.104269 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:36971 every 8 connection(s)
I20260812 11:11:41.105232 5085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:41.106612 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 5085
I20260812 11:11:41.106765 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:41.107090 23943 external_mini_cluster.cc:1428] Setting key ed9cd4888d4cbba73504c271f73592b8
I20260812 11:11:41.108640 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:41.115186 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Bootstrap starting.
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:41.117689 5234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:41.118758 5234 log.cc:824] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:41.120971 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: No bootstrap required, opened a new log
I20260812 11:11:41.122565 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.108525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.23.97.254:36971)
0812 11:11:41.108967 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.108982 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.109018 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:11:41.109664 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.109681 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.109990 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.110270 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.111214 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.112183 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.113413 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.113861 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.113979 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.116868 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.116873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.116885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.116912 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.119502 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.119946 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.119952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.119957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.120097 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.120403 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.120406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.120407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.120891 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.121088 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.121384 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.121669 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260812 11:11:41.124675 5234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } }
I20260812 11:11:41.124943 5234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:41.124993 5234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0278565299a245369695a50050293d15, State: Initialized, Role: FOLLOWER
I20260812 11:11:41.125334 5234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } }
I20260812 11:11:41.125443 5234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:41.125489 5234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:41.125555 5234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:41.126322 5234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } }
I20260812 11:11:41.126598 5234 leader_election.cc:302] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0278565299a245369695a50050293d15; no voters:
I20260812 11:11:41.126844 5234 leader_election.cc:288] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:41.126976 5243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:41.127220 5243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [term 1 LEADER]: Becoming Leader. State: Replica: 0278565299a245369695a50050293d15, State: Running, Role: LEADER
I20260812 11:11:41.127544 5243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } }
I20260812 11:11:41.128052 5234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:41.129365 5246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0278565299a245369695a50050293d15. Latest consensus state: current_term: 1 leader_uuid: "0278565299a245369695a50050293d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } } }
I20260812 11:11:41.129361 5245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0278565299a245369695a50050293d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0278565299a245369695a50050293d15" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 36971 } } }
I20260812 11:11:41.129526 5245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:41.129526 5246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:41.130023 5259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:41.133160 5259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:41.138895 5259 catalog_manager.cc:1347] Generated new cluster ID: 2d044a55c8fb4720b89beb82b09f4e59
I20260812 11:11:41.138969 5259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:41.149236 5259 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:41.150491 5259 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:41.156122 5259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Generated new TSK 0
I20260812 11:11:41.156873 5259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
I20260812 11:11:41.195298 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36971
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:41.306535 5288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:41.306792 5288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:41.306864 5288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:41.310930 5288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:41.311012 5288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:41.311082 5288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:41.311162 5288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:41.316087 5288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:41.317076 5288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:41.318030 5288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:41.327215 5299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.327270 5296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.327316 5288 server_base.cc:1034] running on GCE node
W20260812 11:11:41.327235 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.327841 5288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:41.328519 5288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:41.329676 5288 hybrid_clock.cc:648] HybridClock initialized: now 1786533101329663 us; error 28 us; skew 500 ppm
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:41.332602 5288 init.cc:377] Logged in from keytab as oryx/127.23.97.193@KRBTEST.COM (short username oryx)
I20260812 11:11:41.333400 5288 webserver.cc:460] Webserver started at http://127.23.97.193:46349/ using document root <none> and password file <none>
I20260812 11:11:41.333942 5288 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:41.334035 5288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:41.334273 5288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:41.336180 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "413802e34b0e4fb4afe099a02dd22487"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "7098daf1afe0fce9fd1b2a743071e3cb"
server_key_iv: "72aff433ba6793f2fae53d538ee9bc1c"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.336804 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "413802e34b0e4fb4afe099a02dd22487"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "7098daf1afe0fce9fd1b2a743071e3cb"
server_key_iv: "72aff433ba6793f2fae53d538ee9bc1c"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.340302 5288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:11:41.342842 5316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.343993 5288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:11:41.344131 5288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "413802e34b0e4fb4afe099a02dd22487"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "7098daf1afe0fce9fd1b2a743071e3cb"
server_key_iv: "72aff433ba6793f2fae53d538ee9bc1c"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.344240 5288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:41.357455 5288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:41.360581 5288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:41.360786 5288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:41.362128 5288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:41.362205 5288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.362286 5288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:41.362331 5288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:41.373740 5288 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44683
I20260812 11:11:41.374718 5288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:41.375535 5537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44683 every 8 connection(s)
I20260812 11:11:41.376488 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.362933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:38805 (local address 127.23.97.254:36971)
0812 11:11:41.363063 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.363066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.364105 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 11:11:41.365477 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.365480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.365538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.365623 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.367046 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.367978 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.369253 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.369574 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.369714 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.372550 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.372555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.372557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.372585 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.374182 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.374747 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.374750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.374751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.374860 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.375265 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.375269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.375271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.375524 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.375631 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.376191 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.376295 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:11:41.377203 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.363449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.193:38805)
0812 11:11:41.363916 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.363999 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.365261 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.365811 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.365819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.366324 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.366870 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.366890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.368125 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.368129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.369116 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.369125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.369763 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.369767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.369939 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.370466 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.370493 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.372351 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.374296 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.374301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.374314 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.374636 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.374941 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.374945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.374947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.375128 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.375717 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.375726 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.375954 (+ 228us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.376103 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.376327 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260812 11:11:41.378397 5539 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36971
I20260812 11:11:41.378549 5539 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:41.379007 5539 heartbeater.cc:511] Master 127.23.97.254:36971 requested a full tablet report, sending...
I20260812 11:11:41.380457 5139 ts_manager.cc:194] Registered new tserver with Master: 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683)
I20260812 11:11:41.381395 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 5288
I20260812 11:11:41.381518 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:41.381801 23943 external_mini_cluster.cc:1428] Setting key 5ab2f0db85cad6c3d731005e1a5bc9e1
I20260812 11:11:41.381961 5139 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:38805
I20260812 11:11:41.388422 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.380491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:57445 (local address 127.23.97.254:36971)
0812 11:11:41.380615 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.380619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.380774 (+ 155us) server_negotiation.cc:407] Connection header received
0812 11:11:41.380860 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.380862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.380900 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.380973 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.381380 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.381876 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.382983 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.383302 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.383405 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.384841 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.384845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.384847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.384873 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.386656 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.387138 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.387142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.387144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.387230 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.387565 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.387568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.387570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.387726 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.387803 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.388177 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.388271 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:11:41.388552 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.380344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.193:57445)
0812 11:11:41.380693 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.380705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.380784 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.381000 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.381002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.381162 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.381249 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.381266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.382018 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.382021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.382850 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.382860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.383515 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.383519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.383606 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.384104 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.384118 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.384688 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.386784 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.386787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.386788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.387019 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.387340 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.387347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.387351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.387458 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.387828 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.387834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.387946 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.388068 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.388231 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
I20260812 11:11:41.410645 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36971
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:41.521486 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:41.521705 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:41.521747 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:41.525513 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:41.525556 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:41.525592 5557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:41.525624 5557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:41.530694 5557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:41.531536 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:41.532426 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:41.541843 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.541875 5557 server_base.cc:1034] running on GCE node
W20260812 11:11:41.541772 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.541813 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.542426 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:41.543035 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:41.544229 5557 hybrid_clock.cc:648] HybridClock initialized: now 1786533101544196 us; error 45 us; skew 500 ppm
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:41.547014 5557 init.cc:377] Logged in from keytab as oryx/127.23.97.194@KRBTEST.COM (short username oryx)
I20260812 11:11:41.547857 5557 webserver.cc:460] Webserver started at http://127.23.97.194:43375/ using document root <none> and password file <none>
I20260812 11:11:41.548504 5557 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:41.548604 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:41.548812 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:41.550747 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "639140be97584e26ba4cb9d7cde87b08"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "af35db2fbce396d9611196362859aa5a"
server_key_iv: "3af292c96f994d3557853cc975a46107"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.551301 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "639140be97584e26ba4cb9d7cde87b08"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "af35db2fbce396d9611196362859aa5a"
server_key_iv: "3af292c96f994d3557853cc975a46107"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.554864 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 11:11:41.557255 5585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.558406 5557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:41.558549 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "639140be97584e26ba4cb9d7cde87b08"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "af35db2fbce396d9611196362859aa5a"
server_key_iv: "3af292c96f994d3557853cc975a46107"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.558658 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:41.581856 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:41.584757 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:41.584957 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:41.586378 5557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:41.586452 5557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.586519 5557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:41.586565 5557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.596464 5557 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:43505
I20260812 11:11:41.596799 5810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:43505 every 8 connection(s)
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:41.597549 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:41.601161 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.587388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:54791 (local address 127.23.97.254:36971)
0812 11:11:41.587525 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.587528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.588623 (+ 1095us) server_negotiation.cc:407] Connection header received
0812 11:11:41.589874 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.589881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.589953 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.590061 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.591672 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.592617 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.594219 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.594604 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.594739 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.597672 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.597676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.597678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.597714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.599306 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.599767 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.599769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.599771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.599848 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.600169 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.600171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.600173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.600337 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.600490 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.600920 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.601016 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:11:41.601886 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.587920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.194:54791)
0812 11:11:41.588465 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.588517 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.589577 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.590278 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.590289 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.590739 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.591473 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.591497 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.592761 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.592764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.594077 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.594088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.594790 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.594797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.595014 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.595694 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.595710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.597475 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.599413 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.599418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.599429 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.599677 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.599930 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.599932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.599933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.600060 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.600496 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.600501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.600712 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.600869 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.601101 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260812 11:11:41.603183 5813 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36971
I20260812 11:11:41.603335 5813 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:41.603868 5813 heartbeater.cc:511] Master 127.23.97.254:36971 requested a full tablet report, sending...
I20260812 11:11:41.604995 5139 ts_manager.cc:194] Registered new tserver with Master: 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:41.605808 5139 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:54791
I20260812 11:11:41.606793 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 5557
I20260812 11:11:41.606936 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:41.607287 23943 external_mini_cluster.cc:1428] Setting key 851ff10596c9bcf34b3bbc1c02738070
I20260812 11:11:41.613130 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.605328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:57127 (local address 127.23.97.254:36971)
0812 11:11:41.605450 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.605454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.605638 (+ 184us) server_negotiation.cc:407] Connection header received
0812 11:11:41.605709 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.605711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.605753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.605842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.606368 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.606914 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.607951 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.608316 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.608590 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.609728 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.609735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.609741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.609774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.611365 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.611888 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.611891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.611892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.611962 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.612261 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.612263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.612264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.612529 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.612611 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.612939 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.613028 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":49}
I20260812 11:11:41.613296 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.605266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.194:57127)
0812 11:11:41.605558 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.605569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.605649 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.605861 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.605863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.606103 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.606232 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.606238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.607073 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.607075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.607781 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.607793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.608695 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.608698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.608767 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.609248 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.609259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.609581 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.611480 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.611485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.611489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.611786 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.612051 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.612055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.612059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.612161 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.612628 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.612634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.612763 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.612885 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.613045 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
I20260812 11:11:41.636593 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:36971
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:41.747514 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:41.747823 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:41.747910 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:41.752059 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:41.752141 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:41.752213 5830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:11:41.752276 5830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:41.757223 5830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.97.254:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:41.758205 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:41.759186 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:41.768905 5837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.769014 5836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:41.768934 5839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:41.769297 5830 server_base.cc:1034] running on GCE node
I20260812 11:11:41.769685 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:41.770318 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:41.771507 5830 hybrid_clock.cc:648] HybridClock initialized: now 1786533101771490 us; error 28 us; skew 500 ppm
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:41.774240 5830 init.cc:377] Logged in from keytab as oryx/127.23.97.195@KRBTEST.COM (short username oryx)
I20260812 11:11:41.775101 5830 webserver.cc:460] Webserver started at http://127.23.97.195:38615/ using document root <none> and password file <none>
I20260812 11:11:41.775653 5830 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:41.775758 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:41.775990 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:41.778025 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "771603cef6c74b3283eafa22d5e478d7"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "9cc52cd8247436f30a91ef2d01944770"
server_key_iv: "4ff6f0c15953f1e6d18f5ab0f30f1eef"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.778720 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "771603cef6c74b3283eafa22d5e478d7"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "9cc52cd8247436f30a91ef2d01944770"
server_key_iv: "4ff6f0c15953f1e6d18f5ab0f30f1eef"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.782516 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:11:41.785068 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.786327 5830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:41.786473 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "771603cef6c74b3283eafa22d5e478d7"
format_stamp: "Formatted at 2026-08-12 11:11:41 on dist-test-slave-zpvx"
server_key: "9cc52cd8247436f30a91ef2d01944770"
server_key_iv: "4ff6f0c15953f1e6d18f5ab0f30f1eef"
server_key_version: "encryptionkey@0"
I20260812 11:11:41.786585 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:41.812114 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:41.815176 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:41.815393 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:41.816938 5830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:41.817011 5830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:41.817085 5830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:41.817137 5830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:41.828619 5830 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39815
I20260812 11:11:41.828647 6081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39815 every 8 connection(s)
I20260812 11:11:41.829622 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:41.831776 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.817673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:59807 (local address 127.23.97.254:36971)
0812 11:11:41.817837 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.817843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.818695 (+ 852us) server_negotiation.cc:407] Connection header received
0812 11:11:41.819673 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.819676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.819740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.819847 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.821256 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.822122 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.823546 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.824225 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.824406 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.827636 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.827640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.827645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.827682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.829611 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.830329 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.830336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.830340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.830458 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.830808 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.830810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.830812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.830982 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.831062 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.831551 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.831659 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:11:41.832460 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 5830
I20260812 11:11:41.832605 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:41.832667 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.818060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.195:59807)
0812 11:11:41.818526 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.818577 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.819458 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.820129 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.820140 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.820632 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.821104 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.821118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.822276 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.822280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.823364 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.823377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.824599 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.824605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.824838 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.825454 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.825475 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.827401 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.829768 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.829773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.829786 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.830162 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.830574 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.830577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.830578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.830710 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.831074 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.831079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.831303 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.831456 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.831688 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260812 11:11:41.832957 23943 external_mini_cluster.cc:1428] Setting key b6ef06f20e5e1cd920bbc5072bbe6d5a
I20260812 11:11:41.834085 6083 heartbeater.cc:347] Connected to a master server at 127.23.97.254:36971
I20260812 11:11:41.834277 6083 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:41.834828 6083 heartbeater.cc:511] Master 127.23.97.254:36971 requested a full tablet report, sending...
I20260812 11:11:41.835880 5138 ts_manager.cc:194] Registered new tserver with Master: 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:41.836633 5138 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:59807
I20260812 11:11:41.842999 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.836269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.23.97.195:38331)
0812 11:11:41.836645 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.836662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.836763 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.837033 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.837036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.837264 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:41.837378 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.837384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.838379 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.838382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.838914 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.838922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.839333 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.839335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.839402 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.839902 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:41.839923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:41.840227 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:41.841722 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.841725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.841726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.841912 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.842166 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:41.842168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:41.842169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:41.842242 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:41.842591 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:41.842593 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:41.842668 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.842754 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.842857 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":195,"threads_started":1}
I20260812 11:11:41.843005 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.836353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:38331 (local address 127.23.97.254:36971)
0812 11:11:41.836514 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.836518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.836753 (+ 235us) server_negotiation.cc:407] Connection header received
0812 11:11:41.836846 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.836849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.836917 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.836995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.837501 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.838232 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.839020 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.839231 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.839313 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.840364 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.840410 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.840415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.840438 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.841610 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.842005 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.842008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.842009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.842082 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.842347 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.842349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.842350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.842501 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.842575 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.842821 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.842907 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:11:41.846808 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:11:41.872941 5239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:41.866307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.23.97.254:36971)
0812 11:11:41.866471 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.866475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.866489 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:41.866528 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.866530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.866570 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.866668 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.867512 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.868081 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.869118 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.869353 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.869469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.872765 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52656: BlockingRecv error: recv got EOF from 127.0.0.1:52656 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 11:11:41 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:41.881965 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.874348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.23.97.254:36971)
0812 11:11:41.874504 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.874507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.874585 (+ 78us) server_negotiation.cc:407] Connection header received
0812 11:11:41.874676 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.874678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.874713 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.874779 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:41.875328 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.875806 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.876612 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.876823 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.876901 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.879168 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:41.879186 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:41.879188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:41.879209 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:41.880553 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.880923 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.880925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.880926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.880999 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:41.881225 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:41.881227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:41.881228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:41.881384 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:41.881483 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.881674 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.881760 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:11:41.884714 5138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:41.887581 5138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:41.902335 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.896186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44683 (local address 127.0.0.1:48972)
0812 11:11:41.896943 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.896991 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.897149 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.897890 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.897896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.897921 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:41.898066 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.898074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.899416 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.899418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.900613 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.900624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.901788 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.901792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.901905 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.901944 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.902062 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.902177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":78,"threads_started":1}
I20260812 11:11:41.902577 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.896399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.0.0.1:37956)
0812 11:11:41.896943 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.896993 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.897120 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.898645 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.898647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.898661 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:41.898735 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.898738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.900034 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.900037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.901169 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.901179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.902170 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.902174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.902258 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.902282 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.902334 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.902396 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":112,"threads_started":1}
I20260812 11:11:41.902400 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.896411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.23.97.193:44683)
0812 11:11:41.896684 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.896691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.897073 (+ 382us) server_negotiation.cc:407] Connection header received
0812 11:11:41.897310 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.897316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.897535 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.897661 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:41.898202 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.899287 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.900760 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.901501 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.901565 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.901668 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.902131 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.902239 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260812 11:11:41.903230 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.897254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.23.97.195:39815)
0812 11:11:41.898204 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.898217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.898238 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:11:41.898323 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.898328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.898507 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.898662 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:41.898866 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.899899 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.901305 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.902003 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.902765 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.902957 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.903008 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.903095 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":102,"threads_started":1}
I20260812 11:11:41.903465 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.896957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43505 (local address 127.0.0.1:59660)
0812 11:11:41.898068 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:41.898081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:41.898180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:41.899205 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:41.899208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:41.899227 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:41.899334 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.899340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.900716 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.900719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.901634 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:41.901645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.903077 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.903080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:41.903143 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.903160 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:11:41.903257 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:11:41.903340 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":101,"threads_started":1}
I20260812 11:11:41.903882 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:41.897556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.23.97.194:43505)
0812 11:11:41.897887 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:41.897897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:41.898810 (+ 913us) server_negotiation.cc:407] Connection header received
0812 11:11:41.898891 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:41.898895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:41.899054 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:41.899192 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:41.899476 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.900589 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.901779 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:41.902624 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:41.902863 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:41.903020 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:41.903454 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 11:11:41.903748 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":127,"threads_started":1}
I20260812 11:11:41.905606 5411 tablet_service.cc:1467] Processing CreateTablet for tablet afc582a5c748433fb2700787e3d76302 (DEFAULT_TABLE table=test-table [id=b57fddf489a848f5bde835ebe1c1ef16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:41.906581 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc582a5c748433fb2700787e3d76302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:41.906677 5955 tablet_service.cc:1467] Processing CreateTablet for tablet afc582a5c748433fb2700787e3d76302 (DEFAULT_TABLE table=test-table [id=b57fddf489a848f5bde835ebe1c1ef16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:41.906677 5688 tablet_service.cc:1467] Processing CreateTablet for tablet afc582a5c748433fb2700787e3d76302 (DEFAULT_TABLE table=test-table [id=b57fddf489a848f5bde835ebe1c1ef16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:41.907699 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc582a5c748433fb2700787e3d76302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:41.907720 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc582a5c748433fb2700787e3d76302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:41.912639 6141 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Bootstrap starting.
I20260812 11:11:41.914911 6141 tablet_bootstrap.cc:654] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:41.915378 6144 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Bootstrap starting.
I20260812 11:11:41.915845 6141 log.cc:824] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:41.916409 6142 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Bootstrap starting.
I20260812 11:11:41.917723 6144 tablet_bootstrap.cc:654] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:41.917974 6141 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: No bootstrap required, opened a new log
I20260812 11:11:41.918172 6141 ts_tablet_manager.cc:1397] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:11:41.918180 6142 tablet_bootstrap.cc:654] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:41.918967 6144 log.cc:824] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:41.919014 6142 log.cc:824] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:41.920888 6142 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: No bootstrap required, opened a new log
I20260812 11:11:41.921103 6142 ts_tablet_manager.cc:1397] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:11:41.921809 6144 tablet_bootstrap.cc:492] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: No bootstrap required, opened a new log
I20260812 11:11:41.922010 6144 ts_tablet_manager.cc:1397] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 11:11:41.922806 6141 raft_consensus.cc:348] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.923094 6141 raft_consensus.cc:374] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:41.923156 6141 raft_consensus.cc:729] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Initialized, Role: FOLLOWER
I20260812 11:11:41.923617 6141 consensus_queue.cc:260] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.924434 5539 heartbeater.cc:503] Master 127.23.97.254:36971 was elected leader, sending a full tablet report...
I20260812 11:11:41.924469 6142 raft_consensus.cc:348] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.924762 6142 raft_consensus.cc:374] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:41.924821 6142 raft_consensus.cc:729] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Initialized, Role: FOLLOWER
I20260812 11:11:41.924854 6141 ts_tablet_manager.cc:1428] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 11:11:41.925287 6142 consensus_queue.cc:260] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.926127 6083 heartbeater.cc:503] Master 127.23.97.254:36971 was elected leader, sending a full tablet report...
I20260812 11:11:41.926218 6142 ts_tablet_manager.cc:1428] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 11:11:41.927860 6144 raft_consensus.cc:348] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.928143 6144 raft_consensus.cc:374] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:41.928207 6144 raft_consensus.cc:729] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Initialized, Role: FOLLOWER
I20260812 11:11:41.928637 6144 consensus_queue.cc:260] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:41.929646 6144 ts_tablet_manager.cc:1428] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:11:41.929932 5813 heartbeater.cc:503] Master 127.23.97.254:36971 was elected leader, sending a full tablet report...
I20260812 11:11:42.012149 6151 raft_consensus.cc:482] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.012387 6151 raft_consensus.cc:504] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:42.013532 6151 leader_election.cc:288] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815), 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:42.018713 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.014440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:39675 (local address 127.23.97.194:43505)
0812 11:11:42.014565 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.014569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.014818 (+ 249us) server_negotiation.cc:407] Connection header received
0812 11:11:42.014930 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.014933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.014980 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.015064 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.015333 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.016005 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.017384 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.017987 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.018115 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.018166 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.018498 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.018574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:11:42.018698 6152 raft_consensus.cc:482] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.018761 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.013821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.23.97.193:35609)
0812 11:11:42.014175 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.014189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.014345 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.014650 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.014653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.014677 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.014824 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.014830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.015872 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.015875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.017243 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.017255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.018156 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.018160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.018330 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.018375 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.018455 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.018545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":51,"thread_start_us":113,"threads_started":1}
I20260812 11:11:42.018851 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.013999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:35609 (local address 127.23.97.195:39815)
0812 11:11:42.014201 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.014207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.014311 (+ 104us) server_negotiation.cc:407] Connection header received
0812 11:11:42.014484 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.014487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.014542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.014630 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.014939 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.015732 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.017408 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.018000 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.018115 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.018166 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.018649 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.018720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 11:11:42.018887 6152 raft_consensus.cc:504] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:42.018997 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.014325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43505 (local address 127.23.97.193:39675)
0812 11:11:42.014711 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.014723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.014822 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.015090 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.015093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.015109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.015199 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.015203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.016121 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.016124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.017263 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.017270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.018156 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.018160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.018326 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.018373 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.018455 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.018545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":54,"thread_start_us":126,"threads_started":1}
I20260812 11:11:42.019598 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7" is_pre_election: true
I20260812 11:11:42.019845 5991 raft_consensus.cc:2454] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 0.
I20260812 11:11:42.020108 6152 leader_election.cc:288] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:42.020435 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08" is_pre_election: true
I20260812 11:11:42.020520 5322 leader_election.cc:302] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.020725 5730 raft_consensus.cc:2454] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 0.
I20260812 11:11:42.020846 6151 raft_consensus.cc:2781] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:42.020937 6151 raft_consensus.cc:482] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:42.020996 6151 raft_consensus.cc:3037] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.022421 6151 raft_consensus.cc:504] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:42.022791 6151 leader_election.cc:288] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 election: Requested vote from peers 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815), 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:42.023121 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7"
I20260812 11:11:42.023176 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08"
I20260812 11:11:42.023233 5991 raft_consensus.cc:3037] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.023301 5730 raft_consensus.cc:3037] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.024535 5991 raft_consensus.cc:2454] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 1.
I20260812 11:11:42.024657 5730 raft_consensus.cc:2454] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 1.
I20260812 11:11:42.024869 5322 leader_election.cc:302] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.024982 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.020276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44683 (local address 127.23.97.195:54467)
0812 11:11:42.020542 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.020557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.020635 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.021288 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.021291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.021312 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.021421 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.021428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.022379 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.022382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.023522 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.023534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.024568 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.024572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.024677 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.024711 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.024779 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.024851 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260812 11:11:42.025195 6151 raft_consensus.cc:2781] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:42.025413 6151 raft_consensus.cc:686] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 1 LEADER]: Becoming Leader. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Running, Role: LEADER
I20260812 11:11:42.025449 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.020777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54467 (local address 127.23.97.193:44683)
0812 11:11:42.020934 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.020938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.020952 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:42.020992 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.020996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.021046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.021144 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.021548 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.022236 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.023671 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.024267 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.024424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.024501 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.025236 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.025316 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:11:42.025803 6151 consensus_queue.cc:237] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } }
I20260812 11:11:42.025954 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487" is_pre_election: true
I20260812 11:11:42.026692 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.020738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:40691 (local address 127.23.97.194:43505)
0812 11:11:42.020846 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.020850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.021327 (+ 477us) server_negotiation.cc:407] Connection header received
0812 11:11:42.021513 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.021515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.021560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.021608 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.022493 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.023144 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.024671 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.025286 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.025411 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.025456 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.026526 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.026592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 11:11:42.026712 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.020325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43505 (local address 127.23.97.195:40691)
0812 11:11:42.021228 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.021242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.021398 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.021746 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.021749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.021766 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.022349 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.022361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.023300 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.023303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.024466 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.024480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.025412 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.025416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.025633 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.025661 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.026508 (+ 847us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.026581 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":126,"threads_started":1}
I20260812 11:11:42.027441 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc582a5c748433fb2700787e3d76302" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08" is_pre_election: true
I20260812 11:11:42.027621 5730 raft_consensus.cc:2379] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 771603cef6c74b3283eafa22d5e478d7 in current term 1: Already voted for candidate 413802e34b0e4fb4afe099a02dd22487 in this term.
I20260812 11:11:42.028092 5858 leader_election.cc:302] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 771603cef6c74b3283eafa22d5e478d7; no voters: 413802e34b0e4fb4afe099a02dd22487, 639140be97584e26ba4cb9d7cde87b08
I20260812 11:11:42.028352 6152 raft_consensus.cc:2726] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:11:42.029232 5139 catalog_manager.cc:5495] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "413802e34b0e4fb4afe099a02dd22487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:42.060535 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.056613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.23.97.193:44683)
0812 11:11:42.056771 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.056775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.056791 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:11:42.056954 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.056957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.057012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.057126 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.057442 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.057990 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.058953 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.059203 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.059326 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.059653 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.060078 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.060217 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.060294 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.060347 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:11:42.066442 5730 raft_consensus.cc:1261] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Refusing update from remote peer 413802e34b0e4fb4afe099a02dd22487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.066587 5991 raft_consensus.cc:1261] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Refusing update from remote peer 413802e34b0e4fb4afe099a02dd22487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.067143 6160 consensus_queue.cc:1035] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.067309 6151 consensus_queue.cc:1035] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.075745 6166 mvcc.cc:205] Tried to move back new op lower bound from 7317639586057555968 to 7317639585900855296. Current Snapshot: MvccSnapshot[applied={T|T < 7317639586057555968}]
I20260812 11:11:42.076735 6169 mvcc.cc:205] Tried to move back new op lower bound from 7317639586057555968 to 7317639585900855296. Current Snapshot: MvccSnapshot[applied={T|T < 7317639586057555968}]
I20260812 11:11:42.084280 5138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.084479 5138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.086860 5138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Sending DeleteTablet for 3 replicas of tablet afc582a5c748433fb2700787e3d76302
I20260812 11:11:42.087569 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet afc582a5c748433fb2700787e3d76302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:48972
I20260812 11:11:42.087742 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet afc582a5c748433fb2700787e3d76302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59660
I20260812 11:11:42.088083 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet afc582a5c748433fb2700787e3d76302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:37956
I20260812 11:11:42.088199 6183 tablet_replica.cc:330] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: stopping tablet replica
I20260812 11:11:42.088581 6184 tablet_replica.cc:330] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: stopping tablet replica
I20260812 11:11:42.088546 6183 raft_consensus.cc:2229] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.088981 6183 raft_consensus.cc:2258] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.088999 6184 raft_consensus.cc:2229] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.089301 5138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:11:42.089418 6184 raft_consensus.cc:2258] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 11:11:42.089735 5138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:42.091696 6183 ts_tablet_manager.cc:1905] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.091696 6184 ts_tablet_manager.cc:1905] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.094512 6184 ts_tablet_manager.cc:1918] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.094512 6183 ts_tablet_manager.cc:1918] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.094619 6184 log.cc:1196] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/afc582a5c748433fb2700787e3d76302
I20260812 11:11:42.094619 6183 log.cc:1196] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/afc582a5c748433fb2700787e3d76302
I20260812 11:11:42.094949 6183 ts_tablet_manager.cc:1939] T afc582a5c748433fb2700787e3d76302 P 639140be97584e26ba4cb9d7cde87b08: Deleting consensus metadata
I20260812 11:11:42.094949 6184 ts_tablet_manager.cc:1939] T afc582a5c748433fb2700787e3d76302 P 771603cef6c74b3283eafa22d5e478d7: Deleting consensus metadata
I20260812 11:11:42.096050 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 0c402043b3704531a8d0b332571b53b9 (DEFAULT_TABLE table=test-table [id=546ad749e6ea47ce98185f1106bd0445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.096076 6182 tablet_replica.cc:330] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: stopping tablet replica
I20260812 11:11:42.096340 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c402043b3704531a8d0b332571b53b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.096359 5114 catalog_manager.cc:4841] TS 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505): tablet afc582a5c748433fb2700787e3d76302 (table test-table [id=b57fddf489a848f5bde835ebe1c1ef16]) successfully deleted
I20260812 11:11:42.096486 6182 raft_consensus.cc:2229] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:42.096599 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 0c402043b3704531a8d0b332571b53b9 (DEFAULT_TABLE table=test-table [id=546ad749e6ea47ce98185f1106bd0445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.096803 6182 raft_consensus.cc:2258] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.096832 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c402043b3704531a8d0b332571b53b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.097909 6182 ts_tablet_manager.cc:1905] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.098901 6144 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Bootstrap starting.
I20260812 11:11:42.099280 5114 catalog_manager.cc:4841] TS 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815): tablet afc582a5c748433fb2700787e3d76302 (table test-table [id=b57fddf489a848f5bde835ebe1c1ef16]) successfully deleted
I20260812 11:11:42.099337 5955 tablet_service.cc:1467] Processing CreateTablet for tablet 0c402043b3704531a8d0b332571b53b9 (DEFAULT_TABLE table=test-table [id=546ad749e6ea47ce98185f1106bd0445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.099577 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c402043b3704531a8d0b332571b53b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.099886 6144 tablet_bootstrap.cc:654] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.101933 6141 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Bootstrap starting.
I20260812 11:11:42.101936 6142 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Bootstrap starting.
I20260812 11:11:42.102183 6182 ts_tablet_manager.cc:1918] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.102453 6182 log.cc:1196] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/afc582a5c748433fb2700787e3d76302
I20260812 11:11:42.102787 6182 ts_tablet_manager.cc:1939] T afc582a5c748433fb2700787e3d76302 P 413802e34b0e4fb4afe099a02dd22487: Deleting consensus metadata
I20260812 11:11:42.102886 6141 tablet_bootstrap.cc:654] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.103003 6142 tablet_bootstrap.cc:654] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.104082 6141 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: No bootstrap required, opened a new log
I20260812 11:11:42.104176 6141 ts_tablet_manager.cc:1397] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:42.104530 6144 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: No bootstrap required, opened a new log
I20260812 11:11:42.104564 5113 catalog_manager.cc:4841] TS 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683): tablet afc582a5c748433fb2700787e3d76302 (table test-table [id=b57fddf489a848f5bde835ebe1c1ef16]) successfully deleted
I20260812 11:11:42.104619 6144 ts_tablet_manager.cc:1397] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260812 11:11:42.104988 6141 raft_consensus.cc:348] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.105108 6141 raft_consensus.cc:374] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.105147 6141 raft_consensus.cc:729] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.105077 6144 raft_consensus.cc:348] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.105193 6144 raft_consensus.cc:374] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.105237 6144 raft_consensus.cc:729] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.105264 6141 consensus_queue.cc:260] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.105386 6144 consensus_queue.cc:260] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.105479 6141 ts_tablet_manager.cc:1428] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:42.105680 6144 ts_tablet_manager.cc:1428] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:42.113196 6142 tablet_bootstrap.cc:492] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: No bootstrap required, opened a new log
I20260812 11:11:42.113293 6142 ts_tablet_manager.cc:1397] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260812 11:11:42.113754 6142 raft_consensus.cc:348] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.113865 6142 raft_consensus.cc:374] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.113893 6142 raft_consensus.cc:729] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.114004 6142 consensus_queue.cc:260] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.114228 6142 ts_tablet_manager.cc:1428] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260812 11:11:42.126024 5540 tablet.cc:2369] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:42.145848 6153 raft_consensus.cc:482] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.146003 6153 raft_consensus.cc:504] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.147277 6153 leader_election.cc:288] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.152064 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.147510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44683 (local address 127.23.97.194:41611)
0812 11:11:42.147847 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.147862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.148015 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.148400 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.148406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.148432 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.148559 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.148565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.149450 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.149456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.150658 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.150669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.151591 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.151595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.151715 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.151743 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.151833 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.151913 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260812 11:11:42.152050 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.147643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:41611 (local address 127.23.97.193:44683)
0812 11:11:42.147817 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.147822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.147934 (+ 112us) server_negotiation.cc:407] Connection header received
0812 11:11:42.148202 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.148206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.148269 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.148356 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.148679 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.149335 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.150799 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.151449 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.151578 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.151633 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.151865 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.151938 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:11:42.152204 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.147923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:53227 (local address 127.23.97.195:39815)
0812 11:11:42.148039 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.148043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.148063 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:11:42.148524 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.148527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.148574 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.148643 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:11:42.148939 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.149707 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.151052 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.151474 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.151611 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.151665 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.151995 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.152080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:11:42.152606 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.147594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.23.97.194:53227)
0812 11:11:42.148007 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.148020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.148091 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.148710 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.148712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.148729 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:11:42.148827 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.148832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.149837 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.149841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.150932 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.150940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.151716 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.151720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.151840 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.151864 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.152387 (+ 523us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.152467 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260812 11:11:42.153009 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c402043b3704531a8d0b332571b53b9" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487" is_pre_election: true
I20260812 11:11:42.153151 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c402043b3704531a8d0b332571b53b9" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7" is_pre_election: true
I20260812 11:11:42.153214 5450 raft_consensus.cc:2454] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 0.
I20260812 11:11:42.153304 5991 raft_consensus.cc:2454] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 0.
I20260812 11:11:42.153604 5587 leader_election.cc:302] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 639140be97584e26ba4cb9d7cde87b08; no voters:
I20260812 11:11:42.153891 6153 raft_consensus.cc:2781] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:42.153960 6153 raft_consensus.cc:482] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:42.153999 6153 raft_consensus.cc:3037] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.155323 6153 raft_consensus.cc:504] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.155691 6153 leader_election.cc:288] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 election: Requested vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.156013 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c402043b3704531a8d0b332571b53b9" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487"
I20260812 11:11:42.156136 5450 raft_consensus.cc:3037] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.156044 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c402043b3704531a8d0b332571b53b9" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7"
I20260812 11:11:42.156213 5991 raft_consensus.cc:3037] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.157155 5991 raft_consensus.cc:2454] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 1.
I20260812 11:11:42.157377 5450 raft_consensus.cc:2454] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 1.
I20260812 11:11:42.157455 5590 leader_election.cc:302] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639140be97584e26ba4cb9d7cde87b08, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.157637 6153 raft_consensus.cc:2781] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:42.157820 6153 raft_consensus.cc:686] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 1 LEADER]: Becoming Leader. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Running, Role: LEADER
I20260812 11:11:42.158062 6153 consensus_queue.cc:237] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.160430 5138 catalog_manager.cc:5495] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "639140be97584e26ba4cb9d7cde87b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:11:42 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533101, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.254@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:42.195807 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.186228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.23.97.254:36971)
0812 11:11:42.186385 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.186390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.186524 (+ 134us) server_negotiation.cc:407] Connection header received
0812 11:11:42.186627 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.186630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.186683 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.186781 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:42.187427 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.188194 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.189100 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.189448 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.189569 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.192227 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:42.192230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:42.192232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:42.192255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:42.193830 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:42.194423 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:42.194427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:42.194428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:42.194507 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:42.194842 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:42.194845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:42.194846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:42.195024 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:42.195112 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.195498 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.195665 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":64}
I20260812 11:11:42.195869 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.186095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:36971 (local address 127.0.0.1:52668)
0812 11:11:42.186423 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.186447 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.186545 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.186812 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.186815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.187195 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:42.187291 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.187297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.188330 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.188333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.188958 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.188967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.189599 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.189601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.189669 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.190316 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:42.190328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:42.192048 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:42.193955 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:42.193966 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:42.193972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:42.194316 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:42.194605 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:42.194608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:42.194610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:42.194718 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:42.195141 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:42.195148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:42.195297 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.195427 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.195598 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":150,"threads_started":1}
I20260812 11:11:42.199955 5138 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:52668:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:11:42.200827 5138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:42.206177 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 803a5638a4d9428b93c8e634feb787bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5906f93e8b434294bd45c8522c8d1440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:42.206269 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 803a5638a4d9428b93c8e634feb787bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5906f93e8b434294bd45c8522c8d1440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:42.206408 5955 tablet_service.cc:1467] Processing CreateTablet for tablet 803a5638a4d9428b93c8e634feb787bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5906f93e8b434294bd45c8522c8d1440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:11:42.206609 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803a5638a4d9428b93c8e634feb787bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.206698 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803a5638a4d9428b93c8e634feb787bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.206738 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803a5638a4d9428b93c8e634feb787bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.209787 6142 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: Bootstrap starting.
I20260812 11:11:42.209920 6141 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: Bootstrap starting.
I20260812 11:11:42.210726 6142 tablet_bootstrap.cc:654] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.210960 6141 tablet_bootstrap.cc:654] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.212185 6142 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: No bootstrap required, opened a new log
I20260812 11:11:42.212280 6142 ts_tablet_manager.cc:1397] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:42.212560 6144 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: Bootstrap starting.
I20260812 11:11:42.213150 6142 raft_consensus.cc:348] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.213289 6142 raft_consensus.cc:374] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.213313 6141 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: No bootstrap required, opened a new log
I20260812 11:11:42.213321 6142 raft_consensus.cc:729] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.213395 6141 ts_tablet_manager.cc:1397] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 11:11:42.213706 6144 tablet_bootstrap.cc:654] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.213748 6142 consensus_queue.cc:260] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.213837 6141 raft_consensus.cc:348] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.213948 6141 raft_consensus.cc:374] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.213981 6141 raft_consensus.cc:729] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.214002 6152 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.214148 6152 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.214105 6141 consensus_queue.cc:260] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.214399 6141 ts_tablet_manager.cc:1428] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:42.214788 6160 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.214922 6160 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.216686 6142 ts_tablet_manager.cc:1428] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 11:11:42.218664 6144 tablet_bootstrap.cc:492] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: No bootstrap required, opened a new log
I20260812 11:11:42.218739 6144 ts_tablet_manager.cc:1397] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260812 11:11:42.219135 6144 raft_consensus.cc:348] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.219228 6144 raft_consensus.cc:374] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.219249 6144 raft_consensus.cc:729] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.219374 6144 consensus_queue.cc:260] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.219661 6144 ts_tablet_manager.cc:1428] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:42.219633 6193 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.219755 6193 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 11:11:42.331516 6088 tablet.cc:2369] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:11:42.349289 5814 tablet.cc:2369] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:11:42.363974 6152 raft_consensus.cc:482] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.364099 6152 raft_consensus.cc:504] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.364619 6152 leader_election.cc:288] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:42.365079 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803a5638a4d9428b93c8e634feb787bf" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487" is_pre_election: true
I20260812 11:11:42.365144 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803a5638a4d9428b93c8e634feb787bf" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08" is_pre_election: true
I20260812 11:11:42.365243 5450 raft_consensus.cc:2454] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771603cef6c74b3283eafa22d5e478d7 in term 0.
I20260812 11:11:42.365355 5730 raft_consensus.cc:2454] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771603cef6c74b3283eafa22d5e478d7 in term 0.
I20260812 11:11:42.365643 5855 leader_election.cc:302] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.365849 6152 raft_consensus.cc:2781] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:42.365933 6152 raft_consensus.cc:482] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:42.366008 6152 raft_consensus.cc:3037] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.367070 6152 raft_consensus.cc:504] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.367434 6152 leader_election.cc:288] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 election: Requested vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505)
I20260812 11:11:42.367790 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803a5638a4d9428b93c8e634feb787bf" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487"
I20260812 11:11:42.367825 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803a5638a4d9428b93c8e634feb787bf" candidate_uuid: "771603cef6c74b3283eafa22d5e478d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08"
I20260812 11:11:42.367903 5450 raft_consensus.cc:3037] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.367951 5730 raft_consensus.cc:3037] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.368921 5730 raft_consensus.cc:2454] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771603cef6c74b3283eafa22d5e478d7 in term 1.
I20260812 11:11:42.368937 5450 raft_consensus.cc:2454] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771603cef6c74b3283eafa22d5e478d7 in term 1.
I20260812 11:11:42.369236 5855 leader_election.cc:302] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.369416 6152 raft_consensus.cc:2781] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:42.369637 6152 raft_consensus.cc:686] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [term 1 LEADER]: Becoming Leader. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Running, Role: LEADER
I20260812 11:11:42.369949 6152 consensus_queue.cc:237] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.371166 6208 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: TxnStatusTablet state changed. Reason: New leader 771603cef6c74b3283eafa22d5e478d7. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.371307 6208 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:42.371560 6215 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:11:42.371755 6215 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:11:42.372308 5136 catalog_manager.cc:5495] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195). New cstate: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } health_report { overall_health: HEALTHY } } }
I20260812 11:11:42.408015 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.403790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.0.0.1:37960)
0812 11:11:42.404060 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.404075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.404174 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.404569 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.404571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.404579 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:42.404675 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.404679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.405683 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.405685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.406285 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.406292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.406879 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.406882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.406947 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.406993 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.407771 (+ 778us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.407782 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.407831 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.407900 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 11:11:42.408042 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.403938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.23.97.195:39815)
0812 11:11:42.404088 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.404093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.404115 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:11:42.404319 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.404323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.404433 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.404530 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.404786 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.405570 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.406408 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.406768 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.406873 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.407137 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.407671 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.407788 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.407849 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.407905 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:11:42.411592 5730 raft_consensus.cc:1261] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Refusing update from remote peer 771603cef6c74b3283eafa22d5e478d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.411688 5450 raft_consensus.cc:1261] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Refusing update from remote peer 771603cef6c74b3283eafa22d5e478d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.412134 6152 consensus_queue.cc:1035] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.412312 6208 consensus_queue.cc:1035] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.415176 6160 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: TxnStatusTablet state changed. Reason: New leader 771603cef6c74b3283eafa22d5e478d7. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.415302 6160 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.415783 6193 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: TxnStatusTablet state changed. Reason: New leader 771603cef6c74b3283eafa22d5e478d7. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.415890 6193 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.416719 6208 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.416857 6208 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:42.417856 6208 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.417958 6208 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 771603cef6c74b3283eafa22d5e478d7: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:11:42.418048 6165 mvcc.cc:205] Tried to move back new op lower bound from 7317639587472461824 to 7317639587310067712. Current Snapshot: MvccSnapshot[applied={T|T < 7317639587472461824}]
I20260812 11:11:42.418583 6160 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.418685 6160 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 413802e34b0e4fb4afe099a02dd22487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.419489 6193 tablet_replica.cc:439] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "771603cef6c74b3283eafa22d5e478d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } } }
I20260812 11:11:42.419608 6193 tablet_replica.cc:442] T 803a5638a4d9428b93c8e634feb787bf P 639140be97584e26ba4cb9d7cde87b08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:11:42.427410 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.423536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.23.97.194:43505)
0812 11:11:42.423654 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.423658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.423673 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:42.423802 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.423805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.423866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.423966 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.424164 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.424969 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.425826 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.426164 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.426268 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.426493 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.427027 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.427143 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.427223 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.427278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:11:42.438642 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.434933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:42779 (local address 127.23.97.195:39815)
0812 11:11:42.435121 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.435126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.435257 (+ 131us) server_negotiation.cc:407] Connection header received
0812 11:11:42.435394 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.435397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.435449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.435526 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.436131 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.436694 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.437570 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.437797 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.437908 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.438213 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.438279 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.438362 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.438455 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.438504 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 11:11:42.438675 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.434835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.23.97.194:42779)
0812 11:11:42.435158 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.435173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.435286 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.435549 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.435552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.435560 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:42.436013 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.436019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.436817 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.436822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.437443 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.437452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.437923 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.437926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.438027 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.438069 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.438370 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.438378 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.438443 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.438519 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
I20260812 11:11:42.445698 5450 raft_consensus.cc:1261] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Refusing update from remote peer 639140be97584e26ba4cb9d7cde87b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.445739 5991 raft_consensus.cc:1261] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Refusing update from remote peer 639140be97584e26ba4cb9d7cde87b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.446172 6193 consensus_queue.cc:1035] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.446342 6153 consensus_queue.cc:1035] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.470351 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.466260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:41749 (local address 127.23.97.194:43505)
0812 11:11:42.466416 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.466422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.466723 (+ 301us) server_negotiation.cc:407] Connection header received
0812 11:11:42.466908 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.466911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.466955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.467000 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.467335 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.467826 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.469035 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.469400 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.469503 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.469841 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.469930 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.470024 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.470159 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.470209 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:11:42.470352 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.466134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43505 (local address 127.23.97.195:41749)
0812 11:11:42.466644 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.466659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.466797 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.467131 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.467134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.467141 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:42.467234 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.467240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.467931 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.467935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.468923 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.468934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.469523 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.469527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.469627 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.469678 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.470070 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.470078 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.470146 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.470223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":102,"threads_started":1}
I20260812 11:11:42.495214 5139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.495398 5139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.497834 5139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Sending DeleteTablet for 3 replicas of tablet 0c402043b3704531a8d0b332571b53b9
I20260812 11:11:42.498534 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c402043b3704531a8d0b332571b53b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:48972
I20260812 11:11:42.498669 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c402043b3704531a8d0b332571b53b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:37956
I20260812 11:11:42.498731 6182 tablet_replica.cc:330] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: stopping tablet replica
I20260812 11:11:42.498845 6184 tablet_replica.cc:330] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: stopping tablet replica
I20260812 11:11:42.498853 6182 raft_consensus.cc:2229] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.498960 6182 raft_consensus.cc:2258] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.498951 6184 raft_consensus.cc:2229] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.499060 6184 raft_consensus.cc:2258] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.499323 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c402043b3704531a8d0b332571b53b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59660
I20260812 11:11:42.499490 6183 tablet_replica.cc:330] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: stopping tablet replica
I20260812 11:11:42.499594 6183 raft_consensus.cc:2229] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:42.499869 6183 raft_consensus.cc:2258] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.500648 6183 ts_tablet_manager.cc:1905] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.500792 6184 ts_tablet_manager.cc:1905] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.503692 6183 ts_tablet_manager.cc:1918] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:42.503706 6184 ts_tablet_manager.cc:1918] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:42.503765 6183 log.cc:1196] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/0c402043b3704531a8d0b332571b53b9
I20260812 11:11:42.503779 6184 log.cc:1196] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/0c402043b3704531a8d0b332571b53b9
I20260812 11:11:42.504055 6183 ts_tablet_manager.cc:1939] T 0c402043b3704531a8d0b332571b53b9 P 639140be97584e26ba4cb9d7cde87b08: Deleting consensus metadata
I20260812 11:11:42.504067 6184 ts_tablet_manager.cc:1939] T 0c402043b3704531a8d0b332571b53b9 P 771603cef6c74b3283eafa22d5e478d7: Deleting consensus metadata
I20260812 11:11:42.504925 5114 catalog_manager.cc:4841] TS 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505): tablet 0c402043b3704531a8d0b332571b53b9 (table test-table [id=546ad749e6ea47ce98185f1106bd0445]) successfully deleted
I20260812 11:11:42.505424 5114 catalog_manager.cc:4841] TS 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815): tablet 0c402043b3704531a8d0b332571b53b9 (table test-table [id=546ad749e6ea47ce98185f1106bd0445]) successfully deleted
I20260812 11:11:42.509747 6182 ts_tablet_manager.cc:1905] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.512737 6182 ts_tablet_manager.cc:1918] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:42.512830 6182 log.cc:1196] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/0c402043b3704531a8d0b332571b53b9
I20260812 11:11:42.513125 6182 ts_tablet_manager.cc:1939] T 0c402043b3704531a8d0b332571b53b9 P 413802e34b0e4fb4afe099a02dd22487: Deleting consensus metadata
I20260812 11:11:42.513823 5113 catalog_manager.cc:4841] TS 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683): tablet 0c402043b3704531a8d0b332571b53b9 (table test-table [id=546ad749e6ea47ce98185f1106bd0445]) successfully deleted
Aug 12 11:11:42 dist-test-slave-zpvx krb5kdc[5064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:11:42 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:42 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:11:42.540508 5239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:42.530761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.23.97.254:36971)
0812 11:11:42.530881 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.530885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.532831 (+ 1946us) server_negotiation.cc:407] Connection header received
0812 11:11:42.533032 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.533035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.533089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.533180 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:42.534099 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.534964 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.536210 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.536572 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.536687 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.540331 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52680: BlockingRecv error: recv got EOF from 127.0.0.1:52680 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 11:11:42 dist-test-slave-zpvx krb5kdc[5064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM
I20260812 11:11:42.553383 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.543276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.23.97.254:36971)
0812 11:11:42.543386 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.543390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.543550 (+ 160us) server_negotiation.cc:407] Connection header received
0812 11:11:42.543671 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.543676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.543726 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.543813 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:42.544545 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.545337 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.546453 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.546791 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.546892 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.549820 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:42.549826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:42.549831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:42.549868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:42.551559 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:42.552057 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:42.552060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:42.552062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:42.552151 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:42.552521 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:42.552524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:42.552526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:42.552733 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:42.552857 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.553124 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.553231 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:11:42.556327 5139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:11:42.556876 5139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:42.562079 5955 tablet_service.cc:1467] Processing CreateTablet for tablet c132bfef62f447e5aecd145b0c25452f (DEFAULT_TABLE table=test-table [id=9ede2085094b4b1c8f49713d183c6277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.562053 5411 tablet_service.cc:1467] Processing CreateTablet for tablet c132bfef62f447e5aecd145b0c25452f (DEFAULT_TABLE table=test-table [id=9ede2085094b4b1c8f49713d183c6277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.562399 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c132bfef62f447e5aecd145b0c25452f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.562414 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c132bfef62f447e5aecd145b0c25452f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.564440 6141 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Bootstrap starting.
I20260812 11:11:42.564682 5688 tablet_service.cc:1467] Processing CreateTablet for tablet c132bfef62f447e5aecd145b0c25452f (DEFAULT_TABLE table=test-table [id=9ede2085094b4b1c8f49713d183c6277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.564965 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c132bfef62f447e5aecd145b0c25452f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.565315 6141 tablet_bootstrap.cc:654] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.567157 6144 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Bootstrap starting.
I20260812 11:11:42.567274 6142 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Bootstrap starting.
I20260812 11:11:42.568092 6144 tablet_bootstrap.cc:654] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.568095 6142 tablet_bootstrap.cc:654] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.569387 6142 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: No bootstrap required, opened a new log
I20260812 11:11:42.569480 6142 ts_tablet_manager.cc:1397] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 11:11:42.569741 6141 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: No bootstrap required, opened a new log
I20260812 11:11:42.569832 6141 ts_tablet_manager.cc:1397] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 11:11:42.569962 6142 raft_consensus.cc:348] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.570086 6142 raft_consensus.cc:374] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.570122 6142 raft_consensus.cc:729] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.570241 6141 raft_consensus.cc:348] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.570303 6144 tablet_bootstrap.cc:492] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: No bootstrap required, opened a new log
I20260812 11:11:42.570354 6141 raft_consensus.cc:374] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.570264 6142 consensus_queue.cc:260] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.570387 6141 raft_consensus.cc:729] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.570402 6144 ts_tablet_manager.cc:1397] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:42.570498 6141 consensus_queue.cc:260] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.570591 6142 ts_tablet_manager.cc:1428] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:42.570755 6141 ts_tablet_manager.cc:1428] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:11:42.570865 6144 raft_consensus.cc:348] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.570971 6144 raft_consensus.cc:374] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.571002 6144 raft_consensus.cc:729] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.571118 6144 consensus_queue.cc:260] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.571396 6144 ts_tablet_manager.cc:1428] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:42.595957 6230 raft_consensus.cc:482] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.596076 6230 raft_consensus.cc:504] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.596602 6230 leader_election.cc:288] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.597016 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c132bfef62f447e5aecd145b0c25452f" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487" is_pre_election: true
I20260812 11:11:42.597069 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c132bfef62f447e5aecd145b0c25452f" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7" is_pre_election: true
I20260812 11:11:42.597190 5991 raft_consensus.cc:2454] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 0.
I20260812 11:11:42.597186 5450 raft_consensus.cc:2454] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 0.
I20260812 11:11:42.597507 5590 leader_election.cc:302] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 639140be97584e26ba4cb9d7cde87b08, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.597728 6230 raft_consensus.cc:2781] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:42.597779 6230 raft_consensus.cc:482] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:42.597826 6230 raft_consensus.cc:3037] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.598788 6230 raft_consensus.cc:504] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.599151 6230 leader_election.cc:288] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 election: Requested vote from peers 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.599479 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c132bfef62f447e5aecd145b0c25452f" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413802e34b0e4fb4afe099a02dd22487"
I20260812 11:11:42.599550 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c132bfef62f447e5aecd145b0c25452f" candidate_uuid: "639140be97584e26ba4cb9d7cde87b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7"
I20260812 11:11:42.599598 5450 raft_consensus.cc:3037] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.599692 5991 raft_consensus.cc:3037] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.600804 5450 raft_consensus.cc:2454] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 1.
I20260812 11:11:42.600804 5991 raft_consensus.cc:2454] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639140be97584e26ba4cb9d7cde87b08 in term 1.
I20260812 11:11:42.601222 5587 leader_election.cc:302] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 639140be97584e26ba4cb9d7cde87b08; no voters:
I20260812 11:11:42.601732 6230 raft_consensus.cc:2781] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:42.601837 6230 raft_consensus.cc:686] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 1 LEADER]: Becoming Leader. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Running, Role: LEADER
I20260812 11:11:42.601976 6230 consensus_queue.cc:237] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.603998 5138 catalog_manager.cc:5495] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "639140be97584e26ba4cb9d7cde87b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:42.617532 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.614301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.23.97.194:43505)
0812 11:11:42.614422 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.614427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.614487 (+ 60us) server_negotiation.cc:407] Connection header received
0812 11:11:42.614661 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.614665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.614718 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.614797 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.615048 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.615555 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.616469 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.616703 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.616819 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.617083 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.617167 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.617249 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.617341 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.617394 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:11:42.619794 5991 raft_consensus.cc:1261] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Refusing update from remote peer 639140be97584e26ba4cb9d7cde87b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.619794 5450 raft_consensus.cc:1261] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Refusing update from remote peer 639140be97584e26ba4cb9d7cde87b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.620240 6229 consensus_queue.cc:1035] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.620721 6229 consensus_queue.cc:1035] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.628561 5139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.628701 5139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.630929 5139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Sending DeleteTablet for 3 replicas of tablet c132bfef62f447e5aecd145b0c25452f
I20260812 11:11:42.631455 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet c132bfef62f447e5aecd145b0c25452f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:48972
I20260812 11:11:42.631645 6182 tablet_replica.cc:330] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: stopping tablet replica
I20260812 11:11:42.631764 6182 raft_consensus.cc:2229] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.631884 6182 raft_consensus.cc:2258] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.631989 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet c132bfef62f447e5aecd145b0c25452f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:37956
I20260812 11:11:42.632062 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet c132bfef62f447e5aecd145b0c25452f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59660
I20260812 11:11:42.632164 6184 tablet_replica.cc:330] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: stopping tablet replica
I20260812 11:11:42.632267 6184 raft_consensus.cc:2229] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.632328 6183 tablet_replica.cc:330] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: stopping tablet replica
I20260812 11:11:42.632411 6184 raft_consensus.cc:2258] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.632460 6183 raft_consensus.cc:2229] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:42.632483 5139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:11:42.632645 6183 raft_consensus.cc:2258] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 11:11:42.632867 5139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:11:42.633672 6183 ts_tablet_manager.cc:1905] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.635605 6184 ts_tablet_manager.cc:1905] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.635962 6183 ts_tablet_manager.cc:1918] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.636046 6183 log.cc:1196] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/c132bfef62f447e5aecd145b0c25452f
I20260812 11:11:42.636310 6183 ts_tablet_manager.cc:1939] T c132bfef62f447e5aecd145b0c25452f P 639140be97584e26ba4cb9d7cde87b08: Deleting consensus metadata
I20260812 11:11:42.637033 6182 ts_tablet_manager.cc:1905] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.638237 6184 ts_tablet_manager.cc:1918] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.638316 6184 log.cc:1196] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/c132bfef62f447e5aecd145b0c25452f
I20260812 11:11:42.638574 6184 ts_tablet_manager.cc:1939] T c132bfef62f447e5aecd145b0c25452f P 771603cef6c74b3283eafa22d5e478d7: Deleting consensus metadata
I20260812 11:11:42.639283 5114 catalog_manager.cc:4841] TS 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815): tablet c132bfef62f447e5aecd145b0c25452f (table test-table [id=9ede2085094b4b1c8f49713d183c6277]) successfully deleted
I20260812 11:11:42.639595 6182 ts_tablet_manager.cc:1918] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:11:42.639671 6182 log.cc:1196] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/c132bfef62f447e5aecd145b0c25452f
I20260812 11:11:42.639971 6182 ts_tablet_manager.cc:1939] T c132bfef62f447e5aecd145b0c25452f P 413802e34b0e4fb4afe099a02dd22487: Deleting consensus metadata
I20260812 11:11:42.640484 5411 tablet_service.cc:1467] Processing CreateTablet for tablet b531889155324092a0c5ae25ab6b3563 (DEFAULT_TABLE table=test-table [id=8352143bd0894c9ebb04c616a0c07508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.640643 5113 catalog_manager.cc:4841] TS 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683): tablet c132bfef62f447e5aecd145b0c25452f (table test-table [id=9ede2085094b4b1c8f49713d183c6277]) successfully deleted
I20260812 11:11:42.640776 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b531889155324092a0c5ae25ab6b3563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.641482 5955 tablet_service.cc:1467] Processing CreateTablet for tablet b531889155324092a0c5ae25ab6b3563 (DEFAULT_TABLE table=test-table [id=8352143bd0894c9ebb04c616a0c07508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.641714 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b531889155324092a0c5ae25ab6b3563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.642647 5114 catalog_manager.cc:4841] TS 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505): tablet c132bfef62f447e5aecd145b0c25452f (table test-table [id=9ede2085094b4b1c8f49713d183c6277]) successfully deleted
I20260812 11:11:42.642961 6141 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: Bootstrap starting.
I20260812 11:11:42.643710 6142 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Bootstrap starting.
I20260812 11:11:42.643807 6141 tablet_bootstrap.cc:654] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.644486 6142 tablet_bootstrap.cc:654] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.644516 5688 tablet_service.cc:1467] Processing CreateTablet for tablet b531889155324092a0c5ae25ab6b3563 (DEFAULT_TABLE table=test-table [id=8352143bd0894c9ebb04c616a0c07508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:11:42.644776 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b531889155324092a0c5ae25ab6b3563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:42.644956 6141 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: No bootstrap required, opened a new log
I20260812 11:11:42.645038 6141 ts_tablet_manager.cc:1397] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:42.645530 6142 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: No bootstrap required, opened a new log
I20260812 11:11:42.645484 6141 raft_consensus.cc:348] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.645591 6141 raft_consensus.cc:374] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.645598 6142 ts_tablet_manager.cc:1397] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 11:11:42.645632 6141 raft_consensus.cc:729] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.645752 6141 consensus_queue.cc:260] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.646018 6141 ts_tablet_manager.cc:1428] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:42.645988 6142 raft_consensus.cc:348] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.646092 6142 raft_consensus.cc:374] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.646130 6142 raft_consensus.cc:729] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771603cef6c74b3283eafa22d5e478d7, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.646253 6142 consensus_queue.cc:260] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.646492 6142 ts_tablet_manager.cc:1428] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 11:11:42.647075 6144 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Bootstrap starting.
I20260812 11:11:42.648077 6144 tablet_bootstrap.cc:654] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:42.649325 6144 tablet_bootstrap.cc:492] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: No bootstrap required, opened a new log
I20260812 11:11:42.649430 6144 ts_tablet_manager.cc:1397] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:42.649807 6144 raft_consensus.cc:348] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.649894 6144 raft_consensus.cc:374] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:42.649915 6144 raft_consensus.cc:729] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639140be97584e26ba4cb9d7cde87b08, State: Initialized, Role: FOLLOWER
I20260812 11:11:42.650038 6144 consensus_queue.cc:260] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.650331 6144 ts_tablet_manager.cc:1428] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:11:42.817196 6160 raft_consensus.cc:482] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:11:42.817344 6160 raft_consensus.cc:504] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.817752 6160 leader_election.cc:288] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.818372 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b531889155324092a0c5ae25ab6b3563" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08" is_pre_election: true
I20260812 11:11:42.818372 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b531889155324092a0c5ae25ab6b3563" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7" is_pre_election: true
I20260812 11:11:42.818548 5991 raft_consensus.cc:2454] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 0.
I20260812 11:11:42.818548 5730 raft_consensus.cc:2454] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 0.
I20260812 11:11:42.818881 5322 leader_election.cc:302] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.819092 6160 raft_consensus.cc:2781] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:11:42.819193 6160 raft_consensus.cc:482] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:11:42.819250 6160 raft_consensus.cc:3037] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.820330 6160 raft_consensus.cc:504] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.820703 6160 leader_election.cc:288] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 election: Requested vote from peers 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505), 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815)
I20260812 11:11:42.821082 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b531889155324092a0c5ae25ab6b3563" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639140be97584e26ba4cb9d7cde87b08"
I20260812 11:11:42.821147 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b531889155324092a0c5ae25ab6b3563" candidate_uuid: "413802e34b0e4fb4afe099a02dd22487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771603cef6c74b3283eafa22d5e478d7"
I20260812 11:11:42.821193 5730 raft_consensus.cc:3037] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.821290 5991 raft_consensus.cc:3037] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:42.822157 5991 raft_consensus.cc:2454] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 1.
I20260812 11:11:42.822170 5730 raft_consensus.cc:2454] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413802e34b0e4fb4afe099a02dd22487 in term 1.
I20260812 11:11:42.822443 5322 leader_election.cc:302] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413802e34b0e4fb4afe099a02dd22487, 771603cef6c74b3283eafa22d5e478d7; no voters:
I20260812 11:11:42.822618 6160 raft_consensus.cc:2781] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:42.822816 6160 raft_consensus.cc:686] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 1 LEADER]: Becoming Leader. State: Replica: 413802e34b0e4fb4afe099a02dd22487, State: Running, Role: LEADER
I20260812 11:11:42.822973 6160 consensus_queue.cc:237] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } }
I20260812 11:11:42.824656 5139 catalog_manager.cc:5495] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "413802e34b0e4fb4afe099a02dd22487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413802e34b0e4fb4afe099a02dd22487" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 } health_report { overall_health: UNKNOWN } } }
I20260812 11:11:42.846161 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.842869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.23.97.193:44683)
0812 11:11:42.843092 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.843095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.843107 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:11:42.843144 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.843146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.843185 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.843267 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.843494 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.843993 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.845136 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.845364 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.845438 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.845701 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.845766 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.845818 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.845930 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.845975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260812 11:11:42.856024 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.852745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:44291 (local address 127.23.97.195:39815)
0812 11:11:42.852881 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.852885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.853079 (+ 194us) server_negotiation.cc:407] Connection header received
0812 11:11:42.853201 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.853203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.853243 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.853315 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.853547 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.854007 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.855169 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.855380 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.855458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.855696 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.855756 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.855810 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.855889 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.855931 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:11:42.856053 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.852670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39815 (local address 127.23.97.193:44291)
0812 11:11:42.852989 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.853004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.853112 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.853345 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.853348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.853355 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:42.853450 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.853455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.854134 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.854137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.855040 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.855051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.855477 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.855479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.855539 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.855582 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.855829 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.855834 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.855883 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.855945 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260812 11:11:42.861536 5991 raft_consensus.cc:1261] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Refusing update from remote peer 413802e34b0e4fb4afe099a02dd22487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.861596 5730 raft_consensus.cc:1261] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Refusing update from remote peer 413802e34b0e4fb4afe099a02dd22487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:11:42.861992 6160 consensus_queue.cc:1035] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771603cef6c74b3283eafa22d5e478d7" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.862154 6284 consensus_queue.cc:1035] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639140be97584e26ba4cb9d7cde87b08" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:11:42.878695 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.875346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44683 (local address 127.23.97.195:45821)
0812 11:11:42.875520 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.875536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:42.875634 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:42.875850 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:42.875852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:42.875857 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:11:42.875936 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.875940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.876755 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.876760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.877331 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:42.877339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.877908 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.877911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:42.878011 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.878047 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.878364 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.878371 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:11:42.878447 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 11:11:42.878547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 11:11:42.878670 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.875428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:45821 (local address 127.23.97.193:44683)
0812 11:11:42.875542 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:42.875548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:42.875588 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:11:42.875730 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:42.875733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:42.875782 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:42.875849 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:11:42.876048 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.876642 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.877463 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:42.877785 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:42.877895 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:42.878174 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:11:42.878261 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:11:42.878337 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:42.878454 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 11:11:42.878523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:11:42.898734 5136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.898914 5136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:11:42.900904 5136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0278565299a245369695a50050293d15: Sending DeleteTablet for 3 replicas of tablet b531889155324092a0c5ae25ab6b3563
I20260812 11:11:42.901568 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet b531889155324092a0c5ae25ab6b3563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:48972
I20260812 11:11:42.901635 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet b531889155324092a0c5ae25ab6b3563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59660
I20260812 11:11:42.901628 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet b531889155324092a0c5ae25ab6b3563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:11:42 UTC) from {username='oryx', principal='oryx/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:37956
I20260812 11:11:42.901791 6182 tablet_replica.cc:330] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487: stopping tablet replica
I20260812 11:11:42.901791 6183 tablet_replica.cc:330] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: stopping tablet replica
I20260812 11:11:42.901818 6184 tablet_replica.cc:330] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: stopping tablet replica
I20260812 11:11:42.901898 6182 raft_consensus.cc:2229] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:11:42.901906 6183 raft_consensus.cc:2229] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.901932 6184 raft_consensus.cc:2229] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:11:42.902009 6183 raft_consensus.cc:2258] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.902035 6184 raft_consensus.cc:2258] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.902086 6182 raft_consensus.cc:2258] T b531889155324092a0c5ae25ab6b3563 P 413802e34b0e4fb4afe099a02dd22487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:11:42.902166 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 5288
I20260812 11:11:42.913009 6184 ts_tablet_manager.cc:1905] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.914664 6183 ts_tablet_manager.cc:1905] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:11:42.915688 6184 ts_tablet_manager.cc:1918] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:42.915799 6184 log.cc:1196] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/b531889155324092a0c5ae25ab6b3563
I20260812 11:11:42.916047 6184 ts_tablet_manager.cc:1939] T b531889155324092a0c5ae25ab6b3563 P 771603cef6c74b3283eafa22d5e478d7: Deleting consensus metadata
I20260812 11:11:42.916846 5114 catalog_manager.cc:4841] TS 771603cef6c74b3283eafa22d5e478d7 (127.23.97.195:39815): tablet b531889155324092a0c5ae25ab6b3563 (table test-table [id=8352143bd0894c9ebb04c616a0c07508]) successfully deleted
I20260812 11:11:42.918007 6183 ts_tablet_manager.cc:1918] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:11:42.918107 6183 log.cc:1196] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/b531889155324092a0c5ae25ab6b3563
I20260812 11:11:42.918380 6183 ts_tablet_manager.cc:1939] T b531889155324092a0c5ae25ab6b3563 P 639140be97584e26ba4cb9d7cde87b08: Deleting consensus metadata
I20260812 11:11:42.919114 5114 catalog_manager.cc:4841] TS 639140be97584e26ba4cb9d7cde87b08 (127.23.97.194:43505): tablet b531889155324092a0c5ae25ab6b3563 (table test-table [id=8352143bd0894c9ebb04c616a0c07508]) successfully deleted
W20260812 11:11:42.925124 5113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44683 (error 108)
I20260812 11:11:42.926003 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 5557
I20260812 11:11:42.932920 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:42.925885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44683 (local address 127.0.0.1:49008)
0812 11:11:42.932683 (+ 6798us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:42.932781 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
W20260812 11:11:42.934751 5113 catalog_manager.cc:4584] TS 413802e34b0e4fb4afe099a02dd22487 (127.23.97.193:44683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b531889155324092a0c5ae25ab6b3563: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44683: connect: Connection refused (error 111)
I20260812 11:11:42.943022 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 5830
I20260812 11:11:42.958684 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 5085
2026-08-12T11:11:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3504 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:43 dist-test-slave-zpvx krb5kdc[6309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:43 dist-test-slave-zpvx krb5kdc[6309](info): set up 2 sockets
Aug 12 11:11:43 dist-test-slave-zpvx krb5kdc[6309](info): commencing operation
krb5kdc: starting...
W20260812 11:11:44.291126 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:44 dist-test-slave-zpvx krb5kdc[6309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:44.415400 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35389
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:44.526278 6341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:44.526510 6341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:44.526548 6341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:44.530405 6341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:44.530465 6341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:44.530485 6341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:44.530503 6341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:44.530517 6341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:44.535435 6341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:35389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:44.536298 6341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:44.537289 6341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:44.545363 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:44.545415 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:44.545564 6341 server_base.cc:1034] running on GCE node
W20260812 11:11:44.545688 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:44.546293 6341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:44.547333 6341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:44.548444 6341 hybrid_clock.cc:648] HybridClock initialized: now 1786533104548408 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T11:11:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1584 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:11:44 dist-test-slave-zpvx krb5kdc[6366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:11:44 dist-test-slave-zpvx krb5kdc[6366](info): set up 2 sockets
Aug 12 11:11:44 dist-test-slave-zpvx krb5kdc[6366](info): commencing operation
krb5kdc: starting...
W20260812 11:11:45.917935 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:11:46.049773 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43563
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:46.160503 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:46.160758 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:46.160835 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:46.164856 6384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:46.164907 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:46.164927 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:46.164974 6384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:46.165007 6384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:46.170156 6384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:46.171141 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:46.172149 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:46.180546 6392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.180546 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.180595 6396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:46.180732 6384 server_base.cc:1034] running on GCE node
I20260812 11:11:46.181432 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:46.182467 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:46.183670 6384 hybrid_clock.cc:648] HybridClock initialized: now 1786533106183656 us; error 27 us; skew 500 ppm
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:46.191432 6384 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:46.192332 6384 webserver.cc:460] Webserver started at http://127.23.97.254:45827/ using document root <none> and password file <none>
I20260812 11:11:46.192935 6384 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:46.192999 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:46.193234 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:46.195277 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "b6c31076e5d84b279161ac75a4ec42a0"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "a16f19b2bd18005080c52b493a45332b"
server_key_iv: "7249ca9b8de19a024e3571daee3c1ab1"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.195858 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "b6c31076e5d84b279161ac75a4ec42a0"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "a16f19b2bd18005080c52b493a45332b"
server_key_iv: "7249ca9b8de19a024e3571daee3c1ab1"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.199601 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260812 11:11:46.201961 6413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.203145 6384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:46.203299 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "b6c31076e5d84b279161ac75a4ec42a0"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "a16f19b2bd18005080c52b493a45332b"
server_key_iv: "7249ca9b8de19a024e3571daee3c1ab1"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.203410 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:46.238641 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:46.241485 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:46.241691 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:46.248991 6384 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43563
I20260812 11:11:46.249019 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43563 every 8 connection(s)
I20260812 11:11:46.249928 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:46.252724 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:11:46.256142 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 6384
I20260812 11:11:46.256250 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:11:46.256578 23943 external_mini_cluster.cc:1428] Setting key 8b45339897322a7aaaef0163106f1901
I20260812 11:11:46.258634 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Bootstrap starting.
I20260812 11:11:46.261328 6508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Neither blocks nor log segments found. Creating new log.
I20260812 11:11:46.262199 6508 log.cc:824] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:46.264345 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: No bootstrap required, opened a new log
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:46.268715 6508 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:46.269003 6508 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:11:46.269086 6508 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c31076e5d84b279161ac75a4ec42a0, State: Initialized, Role: FOLLOWER
I20260812 11:11:46.269547 6508 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:46.269716 6508 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:46.269819 6508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:46.269945 6508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:11:46.270771 6508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:46.271142 6508 leader_election.cc:302] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c31076e5d84b279161ac75a4ec42a0; no voters:
I20260812 11:11:46.271517 6508 leader_election.cc:288] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:11:46.271577 6521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:11:46.271772 6521 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 LEADER]: Becoming Leader. State: Replica: b6c31076e5d84b279161ac75a4ec42a0, State: Running, Role: LEADER
I20260812 11:11:46.272120 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:46.272226 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.257989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.23.97.254:43563)
0812 11:11:46.258404 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:46.258424 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:46.258467 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:11:46.259272 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:46.259295 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:46.259700 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:46.260076 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:46.260821 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.262334 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.263392 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.264073 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.264230 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.266868 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:46.266876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:46.266894 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:46.266933 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:46.268928 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.269477 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.269483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.269489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.269632 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.270000 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.270003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.270004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.270432 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:46.270652 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:46.270964 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 11:11:46.271307 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260812 11:11:46.272768 6508 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:46.273926 6522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c31076e5d84b279161ac75a4ec42a0. Latest consensus state: current_term: 1 leader_uuid: "b6c31076e5d84b279161ac75a4ec42a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } } }
I20260812 11:11:46.274052 6522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:46.274451 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c31076e5d84b279161ac75a4ec42a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } } }
I20260812 11:11:46.275115 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: This master's current role is: LEADER
W20260812 11:11:46.277643 6551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:11:46.277710 6551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:11:46.277796 6537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:46.278532 6537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:46.284160 6537 catalog_manager.cc:1347] Generated new cluster ID: 5e848b04528645df92082d35de75145d
I20260812 11:11:46.284242 6537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:46.327495 6537 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:11:46.328660 6537 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:46.333385 6537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Generated new TSK 0
I20260812 11:11:46.334031 6537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:11:46.367687 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:46.480943 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:46.481190 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:46.481231 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:46.485348 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:46.485432 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:46.485525 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:46.490584 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:46.491582 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:46.492656 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:46.502254 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.502403 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.502414 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:46.502535 6558 server_base.cc:1034] running on GCE node
I20260812 11:11:46.503026 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:46.503645 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:46.504809 6558 hybrid_clock.cc:648] HybridClock initialized: now 1786533106504775 us; error 45 us; skew 500 ppm
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:46.507630 6558 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:11:46.508466 6558 webserver.cc:460] Webserver started at http://127.23.97.193:40535/ using document root <none> and password file <none>
I20260812 11:11:46.508968 6558 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:46.509027 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:46.509209 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:46.511123 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "8eaff1667bc34e42ad39b596a0661d8a"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "fd52d89c8ab133c8846efae6f3c63eba"
server_key_iv: "a0219a4a44fad8a84338e936e1bc1058"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.511636 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "8eaff1667bc34e42ad39b596a0661d8a"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "fd52d89c8ab133c8846efae6f3c63eba"
server_key_iv: "a0219a4a44fad8a84338e936e1bc1058"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.515295 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:11:46.517642 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.518803 6558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:46.518939 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "8eaff1667bc34e42ad39b596a0661d8a"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "fd52d89c8ab133c8846efae6f3c63eba"
server_key_iv: "a0219a4a44fad8a84338e936e1bc1058"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.519048 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:46.535648 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:46.538328 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:46.538522 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:46.539057 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:46.539836 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:46.539908 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.539978 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:46.540030 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.548938 6558 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33775
I20260812 11:11:46.548981 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33775 every 8 connection(s)
I20260812 11:11:46.549829 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:46.554193 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 6558
I20260812 11:11:46.554347 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:11:46.554668 23943 external_mini_cluster.cc:1428] Setting key d778f2b6a09b19e2ae44d0ccd9ec1490
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:46.564502 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.551611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:55983 (local address 127.23.97.254:43563)
0812 11:11:46.551747 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:46.551751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:46.552567 (+ 816us) server_negotiation.cc:407] Connection header received
0812 11:11:46.553936 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:46.553940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:46.554025 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:46.554131 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:46.555547 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.556213 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.557393 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.557792 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.557923 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.560558 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:46.560565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:46.560570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:46.560605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:46.562318 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.562817 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.562824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.562829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.562931 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.563281 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.563286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.563291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.563554 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:46.563672 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:46.564127 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 11:11:46.564271 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:46.565178 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.551915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:55983)
0812 11:11:46.552351 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:46.552469 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:46.553723 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:46.554343 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:46.554354 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:46.554785 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:46.555392 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.555417 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.556345 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.556348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.557261 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.557272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.557945 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.557948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.558086 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.558732 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:46.558748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:46.560325 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:46.562430 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.562435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.562445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.562691 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.563033 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.563035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.563036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.563161 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.563664 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:46.563668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:46.563872 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 11:11:46.564010 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 11:11:46.564265 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:11:46.566413 6772 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43563
I20260812 11:11:46.566659 6772 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:46.567135 6772 heartbeater.cc:511] Master 127.23.97.254:43563 requested a full tablet report, sending...
I20260812 11:11:46.568655 6443 ts_manager.cc:194] Registered new tserver with Master: 8eaff1667bc34e42ad39b596a0661d8a (127.23.97.193:33775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:11:46.569980 6443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:55983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:11:46.585532 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:46.699321 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:46.699549 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:46.699594 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:46.703682 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:46.703740 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:46.703796 6778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:46.709164 6778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:46.710032 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:46.710943 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:46.720165 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:46.720237 6778 server_base.cc:1034] running on GCE node
W20260812 11:11:46.720180 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.720266 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:46.720757 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:46.721380 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:46.722585 6778 hybrid_clock.cc:648] HybridClock initialized: now 1786533106722570 us; error 27 us; skew 500 ppm
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:46.725493 6778 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:11:46.726368 6778 webserver.cc:460] Webserver started at http://127.23.97.194:43399/ using document root <none> and password file <none>
I20260812 11:11:46.726922 6778 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:46.726984 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:46.727202 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:46.729419 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "987e6c47570d4635851a3415af1dad50"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "6a2b0d6f43d4e44e3055a206f9fdd394"
server_key_iv: "fcc9630e0d4f21ee416d24e8b8e6fb35"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.730000 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "987e6c47570d4635851a3415af1dad50"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "6a2b0d6f43d4e44e3055a206f9fdd394"
server_key_iv: "fcc9630e0d4f21ee416d24e8b8e6fb35"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.733742 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:11:46.736294 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.737545 6778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:46.737702 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "987e6c47570d4635851a3415af1dad50"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "6a2b0d6f43d4e44e3055a206f9fdd394"
server_key_iv: "fcc9630e0d4f21ee416d24e8b8e6fb35"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.737804 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:46.747470 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:46.750173 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:46.750357 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:46.750900 6778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:46.751670 6778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:46.751716 6778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.751776 6778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:46.751813 6778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.760370 6778 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:39491
I20260812 11:11:46.760423 7016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:39491 every 8 connection(s)
I20260812 11:11:46.761212 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:46.761439 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 6778
I20260812 11:11:46.761586 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:11:46.761960 23943 external_mini_cluster.cc:1428] Setting key 4001274569fece641a7f882cd3d7f9be
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:11:46.775632 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.762963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:38797 (local address 127.23.97.254:43563)
0812 11:11:46.763095 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:46.763152 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:46.764187 (+ 1035us) server_negotiation.cc:407] Connection header received
0812 11:11:46.765586 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:46.765590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:46.765653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:46.765734 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:46.766819 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.767375 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.768524 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.768758 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.768870 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.771656 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:46.771659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:46.771661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:46.771685 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:46.773178 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.773758 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.773764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.773767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.773858 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.774204 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.774207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.774208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.774486 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:46.774567 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:46.775080 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 11:11:46.775447 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:11:46.776188 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.763477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:38797)
0812 11:11:46.763995 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:46.764060 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:46.765386 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:46.765897 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:46.765904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:46.766249 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:46.766682 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.766695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.767502 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.767511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.768356 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.768411 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.768911 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.768914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.769090 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.769649 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:46.769664 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:46.771472 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:46.773303 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.773308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.773320 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.773608 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.773986 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.773988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.773990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.774121 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.774582 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:46.774587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:46.774798 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 11:11:46.774988 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 11:11:46.775358 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":156,"threads_started":1}
I20260812 11:11:46.777424 7018 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43563
I20260812 11:11:46.777655 7018 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:46.778151 7018 heartbeater.cc:511] Master 127.23.97.254:43563 requested a full tablet report, sending...
I20260812 11:11:46.779201 6443 ts_manager.cc:194] Registered new tserver with Master: 987e6c47570d4635851a3415af1dad50 (127.23.97.194:39491)
I20260812 11:11:46.779814 6443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:38797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:11:46.791308 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:46.903776 7032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:46.904033 7032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:46.904109 7032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:46.908198 7032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:46.908279 7032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:46.908380 7032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:46.914088 7032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:46.915083 7032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:46.916070 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:46.925993 7048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.925948 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:46.925948 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:46.926133 7032 server_base.cc:1034] running on GCE node
I20260812 11:11:46.926668 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:46.927311 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:46.928508 7032 hybrid_clock.cc:648] HybridClock initialized: now 1786533106928493 us; error 27 us; skew 500 ppm
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:46.931313 7032 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:11:46.932168 7032 webserver.cc:460] Webserver started at http://127.23.97.195:43231/ using document root <none> and password file <none>
I20260812 11:11:46.932749 7032 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:46.932819 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:46.932979 7032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:11:46.934899 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "d050bcc93c194a88a07d02c1c6550064"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "082ccf30a1204e0458bcacb0414adc98"
server_key_iv: "8f5af04d55cbb4b0d3bbe1af0ebdf01e"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.935403 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "d050bcc93c194a88a07d02c1c6550064"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "082ccf30a1204e0458bcacb0414adc98"
server_key_iv: "8f5af04d55cbb4b0d3bbe1af0ebdf01e"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.939061 7032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:11:46.941452 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.942698 7032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:46.942835 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d050bcc93c194a88a07d02c1c6550064"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "082ccf30a1204e0458bcacb0414adc98"
server_key_iv: "8f5af04d55cbb4b0d3bbe1af0ebdf01e"
server_key_version: "encryptionkey@0"
I20260812 11:11:46.942937 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:46.969208 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:46.972126 7032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:46.972333 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:46.972967 7032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:46.973819 7032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:46.973868 7032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.973908 7032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:46.973924 7032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:46.985823 7032 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39149
I20260812 11:11:46.985906 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39149 every 8 connection(s)
I20260812 11:11:46.986714 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:46.987802 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 7032
I20260812 11:11:46.987964 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:11:46.988258 23943 external_mini_cluster.cc:1428] Setting key 2206e51a8b0a642e7296869a6b60f6b2
Aug 12 11:11:46 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533106, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:11:46.999974 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.988535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54717 (local address 127.23.97.254:43563)
0812 11:11:46.988695 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:46.988700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:46.989680 (+ 980us) server_negotiation.cc:407] Connection header received
0812 11:11:46.991090 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:46.991094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:46.991148 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:46.991229 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:46.992388 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.992927 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.994103 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.994333 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.994444 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.996818 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:46.996821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:46.996824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:46.996847 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:46.998266 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.998723 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.998725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.998726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.998799 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:46.999090 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:46.999092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:46.999094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:46.999250 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:46.999324 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:46.999761 (+ 437us) server_negotiation.cc:299] Negotiation successful
0812 11:11:46.999856 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:11:47.000808 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:46.989008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:54717)
0812 11:11:46.989533 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:46.989586 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:46.990883 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:46.991411 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:46.991418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:46.991770 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:46.992213 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.992226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.993062 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.993066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.993968 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:46.993978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:46.994482 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:46.994486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:46.994657 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:46.995151 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:46.995166 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:46.996669 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:11:46.998366 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.998370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.998380 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.998627 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.998881 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:11:46.998883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:11:46.998885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:11:46.999011 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:11:46.999337 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:11:46.999342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:11:46.999548 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 11:11:46.999694 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 11:11:46.999929 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260812 11:11:47.001997 7274 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43563
I20260812 11:11:47.002250 7274 heartbeater.cc:464] Registering TS with master...
I20260812 11:11:47.002719 7274 heartbeater.cc:511] Master 127.23.97.254:43563 requested a full tablet report, sending...
I20260812 11:11:47.003681 6443 ts_manager.cc:194] Registered new tserver with Master: d050bcc93c194a88a07d02c1c6550064 (127.23.97.195:39149)
I20260812 11:11:47.004233 6443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:54717
I20260812 11:11:47.013815 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
Entry for principal oryx/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab.
WARNING: no policy specified for oryx/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
Entry for principal oryx/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab.
WARNING: no policy specified for oryx/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
Entry for principal oryx/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab.
I20260812 11:11:47.085441 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 6558
I20260812 11:11:47.110033 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 6778
I20260812 11:11:47.133760 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 7032
I20260812 11:11:47.145015 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 6384
I20260812 11:11:47.165815 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43563
--webserver_interface=127.23.97.254
--webserver_port=45827
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:11:47.275689 7290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:47.275974 7290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:47.276043 7290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:47.279950 7290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:11:47.280037 7290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:47.280077 7290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:47.280121 7290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:11:47.280157 7290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:11:47.285320 7290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=45827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:47.286550 7290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:47.287566 7290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:47.296026 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.296154 7290 server_base.cc:1034] running on GCE node
W20260812 11:11:47.296048 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:47.296048 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.296890 7290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:47.297953 7290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:47.299160 7290 hybrid_clock.cc:648] HybridClock initialized: now 1786533107299152 us; error 31 us; skew 500 ppm
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533107, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:47.302045 7290 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:11:47.302950 7290 webserver.cc:460] Webserver started at http://127.23.97.254:45827/ using document root <none> and password file <none>
I20260812 11:11:47.303519 7290 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:47.303637 7290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:47.307504 7290 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:47.309777 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.310986 7290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:47.311143 7290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "b6c31076e5d84b279161ac75a4ec42a0"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "a16f19b2bd18005080c52b493a45332b"
server_key_iv: "7249ca9b8de19a024e3571daee3c1ab1"
server_key_version: "encryptionkey@0"
I20260812 11:11:47.311766 7290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:47.340305 7290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:47.343382 7290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:47.343583 7290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:47.350765 7290 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43563
I20260812 11:11:47.350772 7429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43563 every 8 connection(s)
I20260812 11:11:47.351811 7290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:11:47.352344 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 7290
I20260812 11:11:47.352836 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:33775
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:43563
--webserver_port=40535
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:11:47.357133 7431 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:11:47.358887 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Bootstrap starting.
I20260812 11:11:47.377210 7431 log.cc:824] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:11:47.384052 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:11:47.384526 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Bootstrap complete.
I20260812 11:11:47.389222 7431 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:47.389676 7431 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c31076e5d84b279161ac75a4ec42a0, State: Initialized, Role: FOLLOWER
I20260812 11:11:47.390204 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:47.390367 7431 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:11:47.390460 7431 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:11:47.390594 7431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:11:47.392694 7431 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:47.393047 7431 leader_election.cc:302] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c31076e5d84b279161ac75a4ec42a0; no voters:
I20260812 11:11:47.393447 7431 leader_election.cc:288] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:11:47.393545 7437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:11:47.393784 7437 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [term 2 LEADER]: Becoming Leader. State: Replica: b6c31076e5d84b279161ac75a4ec42a0, State: Running, Role: LEADER
I20260812 11:11:47.394094 7437 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } }
I20260812 11:11:47.394747 7431 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:11:47.395083 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c31076e5d84b279161ac75a4ec42a0. Latest consensus state: current_term: 2 leader_uuid: "b6c31076e5d84b279161ac75a4ec42a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } } }
I20260812 11:11:47.395220 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:47.395586 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6c31076e5d84b279161ac75a4ec42a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c31076e5d84b279161ac75a4ec42a0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43563 } } }
I20260812 11:11:47.395802 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:11:47.395608 7444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:11:47.399434 7444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:11:47.400519 7444 catalog_manager.cc:1259] Loaded cluster ID: 5e848b04528645df92082d35de75145d
I20260812 11:11:47.400606 7444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:11:47.402226 7444 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:11:47.402832 7444 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6c31076e5d84b279161ac75a4ec42a0: Loaded TSK: 0
I20260812 11:11:47.403728 7444 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:11:47.477262 7434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:47.477555 7434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:47.477633 7434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:47.481611 7434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:47.481690 7434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:47.481783 7434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:11:47.486933 7434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.193.keytab
--principal=oryx/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:33775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=40535
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:47.487926 7434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:47.488956 7434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:47.499154 7475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:47.499164 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.499501 7434 server_base.cc:1034] running on GCE node
W20260812 11:11:47.499559 7476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.499915 7434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:47.500567 7434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:47.501775 7434 hybrid_clock.cc:648] HybridClock initialized: now 1786533107501750 us; error 48 us; skew 500 ppm
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:47.504921 7434 init.cc:377] Logged in from keytab as oryx/127.23.97.193@KRBTEST.COM (short username oryx)
I20260812 11:11:47.505940 7434 webserver.cc:460] Webserver started at http://127.23.97.193:40535/ using document root <none> and password file <none>
I20260812 11:11:47.506595 7434 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:47.506697 7434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:47.511694 7434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:11:47.514272 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.515724 7434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:11:47.515856 7434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "8eaff1667bc34e42ad39b596a0661d8a"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "fd52d89c8ab133c8846efae6f3c63eba"
server_key_iv: "a0219a4a44fad8a84338e936e1bc1058"
server_key_version: "encryptionkey@0"
I20260812 11:11:47.516566 7434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:47.531831 7434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:47.535254 7434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:47.535445 7434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:47.536128 7434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:47.537139 7434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:47.537202 7434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.537253 7434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:47.537286 7434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.547583 7434 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33775
I20260812 11:11:47.547674 7707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33775 every 8 connection(s)
I20260812 11:11:47.548763 7434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:11:47.550503 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 7434
I20260812 11:11:47.551014 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:39491
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:43563
--webserver_port=43399
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.565765 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.551129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:37095)
0812 11:11:47.551695 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.551748 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.553305 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.555387 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.555397 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.555901 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.556596 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.556617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.558604 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.558607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.559938 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.559949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.561085 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.561089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.561268 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.561928 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.561946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.564918 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":146,"threads_started":1}
W20260812 11:11:47.566661 7709 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:43563 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.97.254:43563: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
W20260812 11:11:47.567387 7714 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.552706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:37095 (local address 127.23.97.254:43563)
0812 11:11:47.554430 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.554445 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.554478 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:11:47.554682 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.554692 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.554971 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.555230 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.556807 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.558433 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.560076 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.560923 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.561122 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.565155 (+ 4033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:37095: BlockingRecv error: recv got EOF from 127.23.97.193:37095 (error 108)
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":130,"threads_started":1}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.572688 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.567133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:36645)
0812 11:11:47.567261 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.567284 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.567360 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.567996 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.567998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.568121 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.568208 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.568212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.569246 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.569249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.569764 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.569773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.570330 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.570332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.570398 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.570857 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.570867 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.572605 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:11:47.573112 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.567311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:36645 (local address 127.23.97.254:43563)
0812 11:11:47.567760 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.567771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.567785 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:47.567820 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.567823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.567864 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.567955 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.568316 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.569134 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.569870 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.570213 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.570318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.572753 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:36645: BlockingRecv error: recv got EOF from 127.23.97.193:36645 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.578123 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.573198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:36349)
0812 11:11:47.573368 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.573386 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.573452 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.573656 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.573658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.573770 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.573831 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.573835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.574809 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.574811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.575295 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.575302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.575847 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.575850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.575902 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.576333 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.576344 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.578043 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260812 11:11:47.578317 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.573243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:36349 (local address 127.23.97.254:43563)
0812 11:11:47.573372 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.573376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.573428 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:11:47.573524 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.573526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.573563 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.573634 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.573941 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.574706 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.575400 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.575738 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.575835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.578197 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:36349: BlockingRecv error: recv got EOF from 127.23.97.193:36349 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 11:11:47.578382 7709 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:11:47.670234 7713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:47.670467 7713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:47.670545 7713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:47.674413 7713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:47.674464 7713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:47.674518 7713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:11:47.679610 7713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.194.keytab
--principal=oryx/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=43399
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:47.680572 7713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:47.681560 7713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:47.691088 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:47.691118 7726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.691253 7713 server_base.cc:1034] running on GCE node
W20260812 11:11:47.691408 7735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.691763 7713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:47.692408 7713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:47.693593 7713 hybrid_clock.cc:648] HybridClock initialized: now 1786533107693574 us; error 29 us; skew 500 ppm
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:47.696911 7713 init.cc:377] Logged in from keytab as oryx/127.23.97.194@KRBTEST.COM (short username oryx)
I20260812 11:11:47.697767 7713 webserver.cc:460] Webserver started at http://127.23.97.194:43399/ using document root <none> and password file <none>
I20260812 11:11:47.698302 7713 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:47.698378 7713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:47.702135 7713 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260812 11:11:47.704090 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.705325 7713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:11:47.705466 7713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "987e6c47570d4635851a3415af1dad50"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "6a2b0d6f43d4e44e3055a206f9fdd394"
server_key_iv: "fcc9630e0d4f21ee416d24e8b8e6fb35"
server_key_version: "encryptionkey@0"
I20260812 11:11:47.705976 7713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:47.730002 7713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:47.732909 7713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:47.733116 7713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:47.733690 7713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:47.734509 7713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:47.734581 7713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.734659 7713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:47.734714 7713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.744299 7713 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:39491
I20260812 11:11:47.744323 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:39491 every 8 connection(s)
I20260812 11:11:47.745280 7713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:11:47.749874 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 7713
I20260812 11:11:47.750401 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:39149
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:43563
--webserver_port=43231
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:11:47.757387 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.747035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:38461 (local address 127.23.97.254:43563)
0812 11:11:47.747166 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.747169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.747996 (+ 827us) server_negotiation.cc:407] Connection header received
0812 11:11:47.749389 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.749392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.749444 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.749517 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.751065 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.751591 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.753056 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.753452 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.753568 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.757196 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:38461: BlockingRecv error: recv got EOF from 127.23.97.194:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:11:47.757790 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.747348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:38461)
0812 11:11:47.747826 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.747868 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.749175 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.749703 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.749715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.750225 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.750903 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.750928 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.751726 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.751732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.752926 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.752936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.753611 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.753614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.753761 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.754359 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.754375 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.756991 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
W20260812 11:11:47.758406 7978 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:43563 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.97.254:43563: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.764357 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.758898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:36823)
0812 11:11:47.759011 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.759026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.759103 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.759387 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.759389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.759514 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.759598 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.759602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.760701 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.760703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.761222 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.761230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.761865 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.761867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.761930 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.762435 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.762446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.764267 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 11:11:47.764580 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.758996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:36823 (local address 127.23.97.254:43563)
0812 11:11:47.759139 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.759145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.759160 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:47.759203 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.759207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.759277 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.759374 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.759715 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.760592 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.761343 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.761748 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.761881 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.764465 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:36823: BlockingRecv error: recv got EOF from 127.23.97.194:36823 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.770098 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.765003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:54615)
0812 11:11:47.765195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.765215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.765274 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.765530 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.765532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.765650 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.765715 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.765720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.766699 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.766701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.767200 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.767207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.767747 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.767750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.767803 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.768256 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.768267 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.770014 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260812 11:11:47.770329 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.765113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:54615 (local address 127.23.97.254:43563)
0812 11:11:47.765231 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.765235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.765249 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:47.765375 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.765378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.765422 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.765504 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.765819 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.766587 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.767309 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.767642 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.767740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.770194 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:54615: BlockingRecv error: recv got EOF from 127.23.97.194:54615 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 11:11:47.770404 7978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:11:47.867959 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:11:47.868192 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:11:47.868232 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:11:47.872236 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:11:47.872288 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:11:47.872399 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:11:47.877539 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/krb5kdc/oryx_127.23.97.195.keytab
--principal=oryx/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:39149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=43231
--tserver_master_addrs=127.23.97.254:43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:11:47.878520 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:11:47.879530 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:11:47.888916 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:11:47.888935 7994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.889011 7984 server_base.cc:1034] running on GCE node
W20260812 11:11:47.888926 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:11:47.889520 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:11:47.890148 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:11:47.891342 7984 hybrid_clock.cc:648] HybridClock initialized: now 1786533107891311 us; error 45 us; skew 500 ppm
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:11:47.894153 7984 init.cc:377] Logged in from keytab as oryx/127.23.97.195@KRBTEST.COM (short username oryx)
I20260812 11:11:47.895249 7984 webserver.cc:460] Webserver started at http://127.23.97.195:43231/ using document root <none> and password file <none>
I20260812 11:11:47.895987 7984 fs_manager.cc:359] Metadata directory not provided
I20260812 11:11:47.896098 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:11:47.899989 7984 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:11:47.902011 8011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.903239 7984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:11:47.903390 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d050bcc93c194a88a07d02c1c6550064"
format_stamp: "Formatted at 2026-08-12 11:11:46 on dist-test-slave-zpvx"
server_key: "082ccf30a1204e0458bcacb0414adc98"
server_key_iv: "8f5af04d55cbb4b0d3bbe1af0ebdf01e"
server_key_version: "encryptionkey@0"
I20260812 11:11:47.903961 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:11:47.925349 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:11:47.928097 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:11:47.928308 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:11:47.928902 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:11:47.929807 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:11:47.929879 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.929950 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:11:47.930002 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:11:47.939157 7984 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39149
I20260812 11:11:47.939189 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39149 every 8 connection(s)
I20260812 11:11:47.940199 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:11:47.947896 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 7984
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:11:47.951349 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.942002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:42017 (local address 127.23.97.254:43563)
0812 11:11:47.942137 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.942141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.942978 (+ 837us) server_negotiation.cc:407] Connection header received
0812 11:11:47.944344 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.944347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.944426 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.944507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.945630 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.946153 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.947289 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.947510 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.947621 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.951212 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:42017: BlockingRecv error: recv got EOF from 127.23.97.195:42017 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:11:47.951939 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.942372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:42017)
0812 11:11:47.942827 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.942868 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.944114 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.944683 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.944690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.945035 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.945479 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.945493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.946282 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.946286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.947154 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.947165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.947674 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.947679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.947850 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.948523 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.948550 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.951107 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
W20260812 11:11:47.952410 8236 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:43563 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.97.254:43563: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.957537 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.948921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.23.97.254:43563)
0812 11:11:47.949089 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.949095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.949111 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:11:47.949159 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.949164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.949221 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.949320 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.950564 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.951064 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.951998 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.952293 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.952438 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.953818 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:11:47.953824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:11:47.953837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:11:47.953863 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:11:47.955464 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:47.955912 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:47.955918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:47.955923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:47.956047 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:11:47.956337 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:11:47.956341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:11:47.956343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:11:47.956709 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:11:47.956909 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 11:11:47.957202 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 11:11:47.957431 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:11:47.958185 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.952828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:35147)
0812 11:11:47.953061 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.953076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.953159 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.953387 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.953390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.953545 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.953627 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.953632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.954467 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.954472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.955142 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.955152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.955604 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.955608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.955694 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.956137 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.956155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.958057 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172}
W20260812 11:11:47.958278 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.952925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:35147 (local address 127.23.97.254:43563)
0812 11:11:47.953055 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.953059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.953132 (+ 73us) server_negotiation.cc:407] Connection header received
0812 11:11:47.953232 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.953236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.953281 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.953372 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.953740 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.954346 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.955276 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.955491 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.955601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.958158 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:35147: BlockingRecv error: recv got EOF from 127.23.97.195:35147 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:47 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:47.963201 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:47.958690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:54969)
0812 11:11:47.958783 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:47.958796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:47.958861 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:47.959044 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:47.959046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:47.959225 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:47.959306 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.959312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.960122 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.960125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.960738 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:47.960748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.961139 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.961142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:47.961197 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.961622 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:47.961634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:47.963128 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 11:11:47.963378 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:47.958744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54969 (local address 127.23.97.254:43563)
0812 11:11:47.958864 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:47.958868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:47.958883 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:11:47.958916 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:47.958918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:47.958953 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:47.959039 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:47.959421 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.959987 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.960850 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:47.961041 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:47.961115 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:47.963263 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:54969: BlockingRecv error: recv got EOF from 127.23.97.195:54969 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 11:11:47.963407 8236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:48.584462 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:48.578969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:33987)
0812 11:11:48.579248 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:48.579265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:48.579381 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:48.579607 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:48.579609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:48.579800 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:48.579894 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.579900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.580661 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.580663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.581245 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.581254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.581724 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.581728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.581797 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.582402 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:48.582414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:48.584284 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260812 11:11:48.584635 8253 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:48.579097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:33987 (local address 127.23.97.254:43563)
0812 11:11:48.579349 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:48.579352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:48.579365 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:48.579453 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:48.579455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:48.579494 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:48.579566 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:48.580001 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.580566 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.581357 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.581573 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.581665 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.584480 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:33987: BlockingRecv error: recv got EOF from 127.23.97.193:33987 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:48.776692 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:48.771057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:51467)
0812 11:11:48.771366 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:48.771382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:48.771528 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:48.771779 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:48.771781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:48.771972 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:48.772072 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.772078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.772899 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.772902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.773487 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.773496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.773913 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.773916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.773984 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.774476 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:48.774488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:48.776556 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
W20260812 11:11:48.776851 8253 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:48.771165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:51467 (local address 127.23.97.254:43563)
0812 11:11:48.771322 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:48.771327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:48.771459 (+ 132us) server_negotiation.cc:407] Connection header received
0812 11:11:48.771625 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:48.771627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:48.771671 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:48.771737 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:48.772177 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.772780 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.773598 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.773807 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.773883 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.776697 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:51467: BlockingRecv error: recv got EOF from 127.23.97.194:51467 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:48 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:48.969872 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:48.964052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:49163)
0812 11:11:48.964355 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:48.964385 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:48.964475 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:48.964700 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:48.964702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:48.964878 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:48.965019 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.965027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.965941 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.965945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.966720 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:48.966730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.967153 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.967156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:48.967237 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.967713 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:48.967724 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:48.969759 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":132,"threads_started":1}
W20260812 11:11:48.970052 8253 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:48.964158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49163 (local address 127.23.97.254:43563)
0812 11:11:48.964305 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:48.964309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:48.964452 (+ 143us) server_negotiation.cc:407] Connection header received
0812 11:11:48.964540 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:48.964542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:48.964587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:48.964658 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:48.965148 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.965796 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.966834 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:48.967050 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:48.967130 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:48.969892 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:49163: BlockingRecv error: recv got EOF from 127.23.97.195:49163 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:49.590547 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:49.585258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:45425)
0812 11:11:49.585660 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:49.585674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:49.585771 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:49.586020 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:49.586022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:49.586232 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:49.586331 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.586336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.587054 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.587056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.587632 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.587640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.588057 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.588059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.588118 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.588640 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:49.588651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:49.590432 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":90,"threads_started":1}
W20260812 11:11:49.590710 8258 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:49.585404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:45425 (local address 127.23.97.254:43563)
0812 11:11:49.585678 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:49.585682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:49.585725 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:11:49.585853 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:49.585856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:49.585899 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:49.585981 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:49.586436 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.586956 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.587738 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.587959 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.588036 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.590571 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:45425: BlockingRecv error: recv got EOF from 127.23.97.193:45425 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:49.783401 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:49.777548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:52105)
0812 11:11:49.777822 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:49.777845 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:49.777951 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:49.778189 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:49.778191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:49.778416 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:49.778618 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.778628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.779652 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.779656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.780304 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.780314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.780812 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.780815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.780876 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.781369 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:49.781381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:49.783220 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
W20260812 11:11:49.783457 8258 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:49.777670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:52105 (local address 127.23.97.254:43563)
0812 11:11:49.777810 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:49.777814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:49.777915 (+ 101us) server_negotiation.cc:407] Connection header received
0812 11:11:49.778031 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:49.778033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:49.778077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:49.778151 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:49.778772 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.779536 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.780487 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.780706 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.780800 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.783351 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:52105: BlockingRecv error: recv got EOF from 127.23.97.194:52105 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:49 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:49.976423 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:49.970601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:34299)
0812 11:11:49.970916 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:49.970929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:49.971025 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:49.971257 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:49.971259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:49.971443 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:49.971538 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.971543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.972278 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.972280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.972901 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:49.972911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.973420 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.973423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:49.973491 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.974107 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:49.974119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:49.976254 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":137,"threads_started":1}
W20260812 11:11:49.976565 8258 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:49.970769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:34299 (local address 127.23.97.254:43563)
0812 11:11:49.970904 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:49.970908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:49.970997 (+ 89us) server_negotiation.cc:407] Connection header received
0812 11:11:49.971093 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:49.971095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:49.971139 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:49.971218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:49.971639 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.972151 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.973015 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:49.973262 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:49.973344 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:49.976451 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:34299: BlockingRecv error: recv got EOF from 127.23.97.195:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:50.597396 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:50.591352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:43537)
0812 11:11:50.591658 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:50.591672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:50.591770 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:50.592061 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:50.592064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:50.592339 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:50.592550 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.592557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.593610 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.593613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.594177 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.594186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.594703 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.594706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.594769 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.595251 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:50.595263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:50.597239 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":122,"threads_started":1}
W20260812 11:11:50.597512 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:50.591404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:43537 (local address 127.23.97.254:43563)
0812 11:11:50.591653 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:50.591657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:50.591736 (+ 79us) server_negotiation.cc:407] Connection header received
0812 11:11:50.591846 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:50.591848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:50.591909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:50.591988 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:50.592736 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.593502 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.594284 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.594595 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.594673 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.597366 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:43537: BlockingRecv error: recv got EOF from 127.23.97.193:43537 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:50.790024 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:50.784216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:34917)
0812 11:11:50.784575 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:50.784597 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:50.784703 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:50.784957 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:50.784959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:50.785147 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:50.785246 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.785253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.786032 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.786035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.786586 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.786594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.786999 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.787001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.787062 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.787639 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:50.787651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:50.789884 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
W20260812 11:11:50.790134 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:50.784292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:34917 (local address 127.23.97.254:43563)
0812 11:11:50.784461 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:50.784467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:50.784668 (+ 201us) server_negotiation.cc:407] Connection header received
0812 11:11:50.784771 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:50.784773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:50.784821 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:50.784908 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:50.785379 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.785932 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.786690 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.786900 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.786979 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.790020 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:34917: BlockingRecv error: recv got EOF from 127.23.97.194:34917 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:50 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:50.983052 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:50.977291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:56117)
0812 11:11:50.977620 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:50.977643 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:50.977801 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:50.978106 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:50.978108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:50.978331 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:50.978433 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.978438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.979199 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.979204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.979770 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:50.979777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.980245 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.980249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:50.980340 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.980960 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:50.980973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:50.982856 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
W20260812 11:11:50.983085 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:50.977424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:56117 (local address 127.23.97.254:43563)
0812 11:11:50.977589 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:50.977594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:50.977723 (+ 129us) server_negotiation.cc:407] Connection header received
0812 11:11:50.977927 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:50.977930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:50.977987 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:50.978077 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:50.978559 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.979083 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.979894 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:50.980124 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:50.980240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:50.982980 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:56117: BlockingRecv error: recv got EOF from 127.23.97.195:56117 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:51.607723 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:51.598233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:38723)
0812 11:11:51.598513 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:51.598527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:51.598638 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:51.598915 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:51.598917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:51.599115 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:51.599212 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.599218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.599947 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.599949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.600577 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.600587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.601038 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.601042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.601114 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.605373 (+ 4259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:51.605391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:51.607595 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
W20260812 11:11:51.607949 8275 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:51.598289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:38723 (local address 127.23.97.254:43563)
0812 11:11:51.598563 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:51.598566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:51.598591 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:11:51.598732 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:51.598734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:51.598783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:51.598861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:51.599319 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.599852 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.600680 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.600921 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.601004 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.607798 (+ 6794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:38723: BlockingRecv error: recv got EOF from 127.23.97.193:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:51.796684 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:51.790803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:38873)
0812 11:11:51.791178 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:51.791198 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:51.791371 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:51.791653 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:51.791655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:51.791879 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:51.791974 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.791980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.792818 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.792823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.793390 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.793397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.793859 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.793862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.793945 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.794533 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:51.794545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:51.796555 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
W20260812 11:11:51.796890 8275 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:51.790936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:38873 (local address 127.23.97.254:43563)
0812 11:11:51.791127 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:51.791133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:51.791303 (+ 170us) server_negotiation.cc:407] Connection header received
0812 11:11:51.791474 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:51.791478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:51.791535 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:51.791628 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:51.792102 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.792696 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.793512 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.793740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.793854 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.796757 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:38873: BlockingRecv error: recv got EOF from 127.23.97.194:38873 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:51 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:51.989400 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:51.983771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:48787)
0812 11:11:51.984052 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:51.984067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:51.984169 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:51.984497 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:51.984499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:51.984700 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:51.984805 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.984811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.985515 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.985518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.986084 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:51.986092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.986544 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.986547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:51.986611 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.987308 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:51.987321 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:51.989283 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
W20260812 11:11:51.989550 8275 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:51.983899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:48787 (local address 127.23.97.254:43563)
0812 11:11:51.984060 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:51.984064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:51.984118 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:11:51.984276 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:51.984278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:51.984328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:51.984425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:51.984893 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.985415 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.986214 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:51.986436 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:51.986526 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:51.989425 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:48787: BlockingRecv error: recv got EOF from 127.23.97.195:48787 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:52.615155 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:52.608719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:57101)
0812 11:11:52.609047 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:52.609101 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:52.609207 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:52.609575 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:52.609578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:52.609788 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:52.609895 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.609901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.610882 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.610885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.611504 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.611515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.611966 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.611968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.612036 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.612629 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:52.612641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:52.615025 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":136,"threads_started":1}
W20260812 11:11:52.615398 8289 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:52.608825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:57101 (local address 127.23.97.254:43563)
0812 11:11:52.609092 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:52.609096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:52.609175 (+ 79us) server_negotiation.cc:407] Connection header received
0812 11:11:52.609285 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:52.609287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:52.609369 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:52.609488 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:52.610069 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.610744 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.611627 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.611860 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.611940 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.615252 (+ 3312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:57101: BlockingRecv error: recv got EOF from 127.23.97.193:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:52.803361 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:52.797518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:41401)
0812 11:11:52.797834 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:52.797848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:52.797982 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:52.798277 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:52.798280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:52.798517 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:52.798628 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.798635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.799423 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.799428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.800023 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.800031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.800594 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.800607 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.800709 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.801270 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:52.801283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:52.803187 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":134,"threads_started":1}
W20260812 11:11:52.803418 8289 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:52.797645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:41401 (local address 127.23.97.254:43563)
0812 11:11:52.797800 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:52.797805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:52.797932 (+ 127us) server_negotiation.cc:407] Connection header received
0812 11:11:52.798091 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:52.798094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:52.798157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:52.798243 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:52.798752 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.799305 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.800161 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.800467 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.800578 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.803310 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:41401: BlockingRecv error: recv got EOF from 127.23.97.194:41401 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:52 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:52.996934 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:52.990242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:37241)
0812 11:11:52.990730 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:52.990749 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:52.990852 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:52.991094 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:52.991096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:52.991304 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:52.991400 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.991406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.992289 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.992293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.992920 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:52.992931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.993455 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.993458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:52.993530 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.994123 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:52.994136 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:52.996741 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":144,"threads_started":1}
W20260812 11:11:52.997002 8289 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:52.990402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:37241 (local address 127.23.97.254:43563)
0812 11:11:52.990554 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:52.990559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:52.990826 (+ 267us) server_negotiation.cc:407] Connection header received
0812 11:11:52.990939 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:52.990942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:52.990990 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:52.991083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:52.991528 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.992166 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.993059 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:52.993317 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:52.993550 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:52.996879 (+ 3329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:37241: BlockingRecv error: recv got EOF from 127.23.97.195:37241 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:53.621783 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:53.615981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:37143)
0812 11:11:53.616358 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:53.616415 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:53.616539 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:53.616810 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:53.616812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:53.616998 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:53.617096 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:53.617101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.618003 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.618006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:53.618590 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:53.618598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.619090 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.619093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:53.619190 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:53.619722 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:53.619734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:53.621665 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
W20260812 11:11:53.622026 8296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:53.616074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:37143 (local address 127.23.97.254:43563)
0812 11:11:53.616422 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:53.616427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:53.616502 (+ 75us) server_negotiation.cc:407] Connection header received
0812 11:11:53.616631 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:53.616634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:53.616685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:53.616771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:53.617249 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.617841 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.618722 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.618978 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.619071 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:53.621810 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:37143: BlockingRecv error: recv got EOF from 127.23.97.193:37143 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":132,"threads_started":1}
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:53.810881 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:53.804231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:58805)
0812 11:11:53.804577 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:53.804594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:53.804747 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:53.805032 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:53.805034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:53.805295 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:53.805393 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:53.805400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.806191 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.806197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:53.806764 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:53.806772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.807286 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.807290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:53.807373 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:53.808044 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:53.808064 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:53.810695 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
W20260812 11:11:53.810950 8296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:53.804393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:58805 (local address 127.23.97.254:43563)
0812 11:11:53.804549 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:53.804554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:53.804680 (+ 126us) server_negotiation.cc:407] Connection header received
0812 11:11:53.804858 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:53.804861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:53.804918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:53.805007 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:53.805522 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.806061 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.806910 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.807143 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.807282 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:53.810830 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:58805: BlockingRecv error: recv got EOF from 127.23.97.194:58805 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:53 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:54.003523 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:53.997931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:41027)
0812 11:11:53.998213 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:53.998231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:53.998328 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:53.998567 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:53.998569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:53.998763 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:53.998858 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:53.998863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.999591 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:53.999593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.000150 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:54.000157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.000710 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.000713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.000783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.001353 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:54.001366 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:54.003352 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260812 11:11:54.003670 8296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:53.998020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:41027 (local address 127.23.97.254:43563)
0812 11:11:53.998180 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:53.998183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:53.998300 (+ 117us) server_negotiation.cc:407] Connection header received
0812 11:11:53.998403 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:53.998405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:53.998451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:53.998528 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:53.998957 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:53.999496 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.000285 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.000590 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.000686 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.003517 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:41027: BlockingRecv error: recv got EOF from 127.23.97.195:41027 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 11:11:54 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:54 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:54.628737 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:54.622713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:55279)
0812 11:11:54.623180 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:54.623197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:54.623342 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:54.623643 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:54.623647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:54.623937 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:54.624037 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:54.624044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.624900 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.624904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.625474 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:54.625482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.625960 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.625964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.626055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.626560 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:54.626572 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:54.628534 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":85,"threads_started":1}
W20260812 11:11:54.628783 8300 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:54.622774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:55279 (local address 127.23.97.254:43563)
0812 11:11:54.623089 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:54.623098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:54.623289 (+ 191us) server_negotiation.cc:407] Connection header received
0812 11:11:54.623448 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:54.623451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:54.623505 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:54.623604 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:54.624174 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.624784 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.625603 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.625836 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.625944 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.628661 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:55279: BlockingRecv error: recv got EOF from 127.23.97.193:55279 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
Aug 12 11:11:54 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:54 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:54.817572 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:54.811655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:55245)
0812 11:11:54.811972 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:54.811986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:54.812084 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:54.812353 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:54.812355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:54.812577 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:54.812691 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:54.812697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.813446 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.813449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.814012 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:54.814019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.814474 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.814477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:54.814545 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.815095 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:54.815107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:54.817382 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
W20260812 11:11:54.817667 8300 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:54.811727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:55245 (local address 127.23.97.254:43563)
0812 11:11:54.811907 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:54.811911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:54.812059 (+ 148us) server_negotiation.cc:407] Connection header received
0812 11:11:54.812170 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:54.812172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:54.812229 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:54.812308 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:54.812823 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.813343 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.814134 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:54.814362 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:54.814453 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:54.817533 (+ 3080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:55245: BlockingRecv error: recv got EOF from 127.23.97.194:55245 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:55.012848 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:55.004480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:55945)
0812 11:11:55.006852 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:55.006866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:55.007016 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:55.007299 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:55.007302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:55.007522 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:55.007680 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.007693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.008538 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.008544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.009191 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.009204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.009783 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.009787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.009874 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.010522 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:55.010537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:55.012662 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2293,"thread_start_us":2124,"threads_started":1}
W20260812 11:11:55.012887 8300 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:55.004550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:55945 (local address 127.23.97.254:43563)
0812 11:11:55.004700 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:55.004704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:55.006956 (+ 2252us) server_negotiation.cc:407] Connection header received
0812 11:11:55.007133 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:55.007136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:55.007184 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:55.007275 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:55.007807 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.008422 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.009333 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.009657 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.009776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.012786 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:55945: BlockingRecv error: recv got EOF from 127.23.97.195:55945 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:55.635107 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:55.629494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:60575)
0812 11:11:55.629811 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:55.629824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:55.629920 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:55.630178 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:55.630180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:55.630367 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:55.630465 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.630470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.631263 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.631266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.631831 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.631839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.632291 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.632293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.632357 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.632872 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:55.632883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:55.634930 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
W20260812 11:11:55.635241 8308 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:55.629675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:60575 (local address 127.23.97.254:43563)
0812 11:11:55.629899 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:55.629903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:55.629917 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:11:55.630009 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:55.630011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:55.630054 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:55.630125 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:55.630607 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.631159 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.631941 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.632187 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.632268 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.635095 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:60575: BlockingRecv error: recv got EOF from 127.23.97.193:60575 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:55 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:55.824271 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:55.818484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:48199)
0812 11:11:55.818760 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:55.818790 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:55.818932 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:55.819206 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:55.819209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:55.819398 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:55.819499 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.819505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.820493 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.820497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.821073 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:55.821081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.821546 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.821549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:55.821617 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.822201 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:55.822214 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:55.824153 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
W20260812 11:11:55.824435 8308 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:55.818541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:48199 (local address 127.23.97.254:43563)
0812 11:11:55.818699 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:55.818704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:55.818904 (+ 200us) server_negotiation.cc:407] Connection header received
0812 11:11:55.819030 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:55.819033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:55.819084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:55.819161 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:55.819628 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.820316 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.821193 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:55.821413 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:55.821545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:55.824294 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:48199: BlockingRecv error: recv got EOF from 127.23.97.194:48199 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:56.021229 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:56.013945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:60255)
0812 11:11:56.014254 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:56.014268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:56.014376 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:56.014644 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:56.014649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:56.014956 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:56.015118 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.015128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.016217 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.016223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.017425 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.017441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.017916 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.017921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.018044 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.018653 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:56.018673 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:56.021022 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260812 11:11:56.021247 8308 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:56.014008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:60255 (local address 127.23.97.254:43563)
0812 11:11:56.014164 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:56.014173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:56.014341 (+ 168us) server_negotiation.cc:407] Connection header received
0812 11:11:56.014446 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:56.014448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:56.014500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:56.014578 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:56.015273 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.016117 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.017566 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.017784 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.017866 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.021146 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:60255: BlockingRecv error: recv got EOF from 127.23.97.195:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:56.646368 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:56.635954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:43083)
0812 11:11:56.636351 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:56.636405 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:56.636524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:56.636777 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:56.636779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:56.636969 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:56.637071 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.637076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.637862 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.637865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.638416 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.638424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.638894 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.638898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.638978 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.644135 (+ 5157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:56.644148 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:56.646186 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":199,"threads_started":1}
W20260812 11:11:56.646456 8315 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:56.636088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:43083 (local address 127.23.97.254:43563)
0812 11:11:56.636351 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:56.636355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:56.636526 (+ 171us) server_negotiation.cc:407] Connection header received
0812 11:11:56.636599 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:56.636601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:56.636655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:56.636744 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:56.637186 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.637762 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.638523 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.638767 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.638851 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.646323 (+ 7472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:43083: BlockingRecv error: recv got EOF from 127.23.97.193:43083 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:56 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:56.831064 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:56.825251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:38143)
0812 11:11:56.825627 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:56.825640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:56.825780 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:56.826079 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:56.826082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:56.826301 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:56.826395 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.826401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.827206 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.827211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.827782 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:56.827790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.828258 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.828261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:56.828349 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.828980 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:56.828999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:56.830889 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
W20260812 11:11:56.831135 8315 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:56.825382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:38143 (local address 127.23.97.254:43563)
0812 11:11:56.825570 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:56.825576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:56.825716 (+ 140us) server_negotiation.cc:407] Connection header received
0812 11:11:56.825909 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:56.825912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:56.825959 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:56.826048 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:56.826516 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.827067 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.827906 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:56.828139 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:56.828252 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:56.831017 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:38143: BlockingRecv error: recv got EOF from 127.23.97.194:38143 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:57.027576 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:57.022157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:49663)
0812 11:11:57.022477 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:57.022494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:57.022624 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:57.022932 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:57.022934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:57.023127 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:57.023222 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.023227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.023947 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.023950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.024561 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.024570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.024990 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.024993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.025056 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.025558 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:57.025570 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:57.027463 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":129,"threads_started":1}
W20260812 11:11:57.027767 8315 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:57.022342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49663 (local address 127.23.97.254:43563)
0812 11:11:57.022475 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:57.022481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:57.022581 (+ 100us) server_negotiation.cc:407] Connection header received
0812 11:11:57.022752 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:57.022755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:57.022817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:57.022891 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:57.023325 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.023842 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.024676 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.024893 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.024972 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.027627 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:49663: BlockingRecv error: recv got EOF from 127.23.97.195:49663 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:57.653178 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:57.647310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:59925)
0812 11:11:57.647617 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:57.647631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:57.647723 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:57.647969 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:57.647971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:57.648152 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:57.648274 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.648279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.649119 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.649122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.649707 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.649715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.650291 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.650295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.650389 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.651017 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:57.651030 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:57.653004 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
W20260812 11:11:57.653303 8328 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:57.647409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:59925 (local address 127.23.97.254:43563)
0812 11:11:57.647693 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:57.647697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:57.647710 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:57.647807 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:57.647809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:57.647851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:57.647927 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:57.648428 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.648971 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.649869 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.650163 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.650285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.653142 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:59925: BlockingRecv error: recv got EOF from 127.23.97.193:59925 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:57 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:57.837854 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:57.831909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:58515)
0812 11:11:57.832263 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:57.832276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:57.832431 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:57.832728 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:57.832732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:57.832945 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:57.833040 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.833046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.833811 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.833816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.834470 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:57.834480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.835026 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.835029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:57.835120 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.835677 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:57.835690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:57.837665 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
W20260812 11:11:57.837916 8328 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:57.832014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:58515 (local address 127.23.97.254:43563)
0812 11:11:57.832197 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:57.832202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:57.832362 (+ 160us) server_negotiation.cc:407] Connection header received
0812 11:11:57.832542 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:57.832546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:57.832612 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:57.832699 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:57.833166 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.833700 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.834593 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:57.834909 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:57.835021 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:57.837805 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:58515: BlockingRecv error: recv got EOF from 127.23.97.194:58515 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:58.034468 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:58.028659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:56285)
0812 11:11:58.028982 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:58.029000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:58.029159 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:58.029457 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:58.029460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:58.029679 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:58.029780 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.029786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.030564 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.030569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.031138 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.031146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.031622 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.031625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.031728 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.032304 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:58.032316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:58.034276 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
W20260812 11:11:58.034507 8328 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:58.028858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:56285 (local address 127.23.97.254:43563)
0812 11:11:58.028982 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:58.028988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:58.029095 (+ 107us) server_negotiation.cc:407] Connection header received
0812 11:11:58.029285 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:58.029288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:58.029342 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:58.029432 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:58.029917 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.030452 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.031279 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.031502 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.031606 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.034404 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:56285: BlockingRecv error: recv got EOF from 127.23.97.195:56285 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:58.659364 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:58.654019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:44623)
0812 11:11:58.654307 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:58.654320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:58.654411 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:58.654665 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:58.654667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:58.654870 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:58.654968 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.654973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.655710 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.655712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.656276 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.656285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.656737 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.656740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.656819 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.657404 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:58.657416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:58.659223 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260812 11:11:58.659488 8336 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:58.654080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:44623 (local address 127.23.97.254:43563)
0812 11:11:58.654356 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:58.654360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:58.654373 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:11:58.654499 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:58.654501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:58.654545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:58.654623 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:58.655070 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.655603 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.656404 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.656635 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.656723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.659362 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:44623: BlockingRecv error: recv got EOF from 127.23.97.193:44623 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:58 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:58.844134 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:58.838643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:51005)
0812 11:11:58.838932 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:58.838946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:58.839057 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:58.839280 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:58.839282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:58.839473 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:58.839573 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.839578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.840316 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.840319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.840948 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:58.840957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.841367 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.841369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:58.841430 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.841906 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:58.841918 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:58.843957 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
W20260812 11:11:58.844237 8336 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:58.838730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:51005 (local address 127.23.97.254:43563)
0812 11:11:58.838877 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:58.838881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:58.839025 (+ 144us) server_negotiation.cc:407] Connection header received
0812 11:11:58.839126 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:58.839128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:58.839174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:58.839244 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:58.839675 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.840210 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.841056 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:58.841267 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:58.841343 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:58.844103 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:51005: BlockingRecv error: recv got EOF from 127.23.97.194:51005 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:59.041412 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:59.035549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:38419)
0812 11:11:59.035836 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:59.035850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:59.035950 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:59.036190 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:59.036192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:59.036448 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:59.036555 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.036561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.037278 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.037280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.038119 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.038132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.038743 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.038747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.038848 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.039361 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:59.039373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:59.041294 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
W20260812 11:11:59.041648 8336 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:59.035642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:38419 (local address 127.23.97.254:43563)
0812 11:11:59.035788 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:59.035792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:59.035919 (+ 127us) server_negotiation.cc:407] Connection header received
0812 11:11:59.036030 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:59.036032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:59.036078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:59.036147 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:59.036662 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.037183 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.038273 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.038614 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.038731 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.041520 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:38419: BlockingRecv error: recv got EOF from 127.23.97.195:38419 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:59.666285 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:59.660288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:43861)
0812 11:11:59.660655 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:59.660672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:59.660823 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:59.661114 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:59.661117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:59.661344 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:59.661440 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.661445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.662227 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.662233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.662805 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.662813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.663294 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.663298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.663388 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.663882 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:59.663894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:59.666161 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":148,"threads_started":1}
W20260812 11:11:59.666427 8340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:59.660344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:43861 (local address 127.23.97.254:43563)
0812 11:11:59.660629 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:59.660633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:59.660764 (+ 131us) server_negotiation.cc:407] Connection header received
0812 11:11:59.660933 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:59.660936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:59.660991 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:59.661090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:59.661565 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.662114 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.662928 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.663167 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.663282 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.666298 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:43861: BlockingRecv error: recv got EOF from 127.23.97.193:43861 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:11:59 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:11:59.850420 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:11:59.844875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:45881)
0812 11:11:59.845141 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 11:11:59.845155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:11:59.845263 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:11:59.845494 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:11:59.845496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:11:59.845687 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:11:59.845788 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.845794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.846521 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.846524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.847077 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:11:59.847085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.847635 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.847638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:11:59.847732 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.848349 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:11:59.848396 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:11:59.850304 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260812 11:11:59.850593 8340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:11:59.845003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:45881 (local address 127.23.97.254:43563)
0812 11:11:59.845144 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:11:59.845149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:11:59.845228 (+ 79us) server_negotiation.cc:407] Connection header received
0812 11:11:59.845339 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:11:59.845341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:11:59.845386 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:11:59.845460 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:11:59.845886 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.846411 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.847273 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:11:59.847493 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:11:59.847632 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:11:59.850438 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:45881: BlockingRecv error: recv got EOF from 127.23.97.194:45881 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:00.048171 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:00.042314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:41435)
0812 11:12:00.042633 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:00.042649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:00.042792 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:00.043078 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:00.043081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:00.043329 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:00.043457 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.043472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.044242 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.044247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.044915 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.044925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.045395 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.045398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.045484 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.046005 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:00.046017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:00.047991 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
W20260812 11:12:00.048210 8340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:00.042373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:41435 (local address 127.23.97.254:43563)
0812 11:12:00.042548 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:00.042554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:00.042752 (+ 198us) server_negotiation.cc:407] Connection header received
0812 11:12:00.042895 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:00.042897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:00.042947 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:00.043048 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:00.043592 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.044124 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.045043 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.045275 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.045388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.048109 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:41435: BlockingRecv error: recv got EOF from 127.23.97.195:41435 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:00.672791 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:00.667142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:49209)
0812 11:12:00.667440 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:00.667455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:00.667547 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:00.667798 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:00.667800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:00.668008 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:00.668105 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.668110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.668908 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.668911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.669505 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.669514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.670036 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.670040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.670144 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.670743 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:00.670756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:00.672649 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
W20260812 11:12:00.672971 8350 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:00.667288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:49209 (local address 127.23.97.254:43563)
0812 11:12:00.667529 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:00.667533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:00.667547 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:12:00.667632 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:00.667634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:00.667677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:00.667748 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:00.668240 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.668801 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.669656 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.669904 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.670018 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.672795 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:49209: BlockingRecv error: recv got EOF from 127.23.97.193:49209 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:00 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:00.856927 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:00.851166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:48595)
0812 11:12:00.851460 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:00.851474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:00.851575 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:00.851814 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:00.851817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:00.852009 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:00.852113 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.852119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.852868 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.852871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.853435 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:00.853445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.853875 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.853878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:00.853948 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.854448 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:00.854460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:00.856755 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
W20260812 11:12:00.857007 8350 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:00.851300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:48595 (local address 127.23.97.254:43563)
0812 11:12:00.851445 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:00.851449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:00.851544 (+ 95us) server_negotiation.cc:407] Connection header received
0812 11:12:00.851658 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:00.851660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:00.851704 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:00.851771 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:00.852218 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.852749 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.853543 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:00.853768 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:00.853849 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:00.856893 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:48595: BlockingRecv error: recv got EOF from 127.23.97.194:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:01.055022 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:01.049097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:54385)
0812 11:12:01.049377 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:01.049391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:01.049496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:01.049745 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:01.049747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:01.049949 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:01.050047 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.050052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.051050 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.051056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.051741 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.051751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.052307 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.052310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.052400 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.052969 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:01.052981 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:01.054905 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
W20260812 11:12:01.055218 8350 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:01.049213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54385 (local address 127.23.97.254:43563)
0812 11:12:01.049367 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:01.049375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:01.049464 (+ 89us) server_negotiation.cc:407] Connection header received
0812 11:12:01.049577 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:01.049579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:01.049626 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:01.049708 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:01.050200 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.050914 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.051886 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.052183 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.052268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.055052 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:54385: BlockingRecv error: recv got EOF from 127.23.97.195:54385 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:01.683796 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:01.673596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:47843)
0812 11:12:01.674100 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:01.674114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:01.674238 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:01.674541 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:01.674544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:01.674758 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:01.674858 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.674864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.675579 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.675581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.676169 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.676177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.676659 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.676662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.676740 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.681770 (+ 5030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:01.681782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:01.683651 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":101,"threads_started":1}
W20260812 11:12:01.683926 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:01.673648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:47843 (local address 127.23.97.254:43563)
0812 11:12:01.674003 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:01.674007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:01.674187 (+ 180us) server_negotiation.cc:407] Connection header received
0812 11:12:01.674320 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:01.674322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:01.674398 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:01.674495 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:01.674963 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.675484 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.676274 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.676556 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.676636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.683794 (+ 7158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:47843: BlockingRecv error: recv got EOF from 127.23.97.193:47843 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:01 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:01.863143 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:01.857752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:48193)
0812 11:12:01.858038 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:01.858051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:01.858149 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:01.858400 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:01.858402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:01.858594 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:01.858693 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.858698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.859433 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.859436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.859999 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:01.860007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.860508 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.860510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:01.860572 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.861043 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:01.861055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:01.863006 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
W20260812 11:12:01.863270 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:01.857813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:48193 (local address 127.23.97.254:43563)
0812 11:12:01.857956 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:01.857960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:01.858120 (+ 160us) server_negotiation.cc:407] Connection header received
0812 11:12:01.858228 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:01.858231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:01.858290 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:01.858358 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:01.858807 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.859337 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.860124 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:01.860405 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:01.860486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:01.863131 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:48193: BlockingRecv error: recv got EOF from 127.23.97.194:48193 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.195@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:02.061730 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:02.055732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.195:34085)
0812 11:12:02.056087 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:02.056104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:02.056243 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:02.056551 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:02.056554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:02.056793 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:02.056890 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.056896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.057684 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.057691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.058285 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.058294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.058770 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.058774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.058866 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.059388 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:02.059400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:02.061603 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":98,"threads_started":1}
W20260812 11:12:02.061877 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:02.055857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:34085 (local address 127.23.97.254:43563)
0812 11:12:02.056101 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:02.056108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:02.056191 (+ 83us) server_negotiation.cc:407] Connection header received
0812 11:12:02.056357 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:02.056361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:02.056434 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:02.056525 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:02.057022 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.057563 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.058416 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.058647 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.058755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.061757 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.195:34085: BlockingRecv error: recv got EOF from 127.23.97.195:34085 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.193@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:02.692355 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:02.684815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.193:60231)
0812 11:12:02.685235 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:02.685252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:02.685380 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:02.685728 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:02.685733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:02.686002 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:02.686150 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.686162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.687438 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.687443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.688519 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.688533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.689211 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.689215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.689320 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.689926 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:02.689941 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:02.692235 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":102,"threads_started":1}
W20260812 11:12:02.692564 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:02.684917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:60231 (local address 127.23.97.254:43563)
0812 11:12:02.685228 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:02.685235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:02.685333 (+ 98us) server_negotiation.cc:407] Connection header received
0812 11:12:02.685510 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:02.685515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:02.685575 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:02.685683 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:02.686306 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.687293 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.688649 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.689018 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.689136 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.692420 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.193:60231: BlockingRecv error: recv got EOF from 127.23.97.193:60231 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:12:02 dist-test-slave-zpvx krb5kdc[6366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.97.194@KRBTEST.COM for oryx/127.23.97.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:12:02.869915 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:02.864193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43563 (local address 127.23.97.194:57853)
0812 11:12:02.864520 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:02.864533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:02.864665 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:02.864960 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:02.864962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:02.865184 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:02.865279 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.865285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.866051 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.866056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.866638 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:02.866647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.867125 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.867129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:02.867220 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.867742 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:02.867755 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:02.869785 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.97.254:43563: Server oryx/127.23.97.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":117,"threads_started":1}
W20260812 11:12:02.870090 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:02.864264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:57853 (local address 127.23.97.254:43563)
0812 11:12:02.864451 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:02.864454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:02.864619 (+ 165us) server_negotiation.cc:407] Connection header received
0812 11:12:02.864776 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:02.864779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:02.864841 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:02.864932 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:02.865403 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.865934 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.866766 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:02.866997 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:02.867109 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:02.869926 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.97.194:57853: BlockingRecv error: recv got EOF from 127.23.97.194:57853 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:12:02.956480 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 7434
I20260812 11:12:02.985450 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 7713
I20260812 11:12:02.998243 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 7984
I20260812 11:12:03.009384 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 7290
2026-08-12T11:12:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18458 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T11:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:12:03Z Disabled control of system clock
I20260812 11:12:03.053730 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35323
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:12:03.166631 8381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:03.166910 8381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:03.166975 8381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:03.170984 8381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:12:03.171069 8381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:03.171108 8381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:03.171159 8381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:12:03.171198 8381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:12:03.175894 8381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:35323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:03.177021 8381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:03.178399 8381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:03.186883 8390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.186909 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.186924 8397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:03.187288 8381 server_base.cc:1034] running on GCE node
I20260812 11:12:03.187884 8381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:03.189013 8381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:03.190184 8381 hybrid_clock.cc:648] HybridClock initialized: now 1786533123190144 us; error 53 us; skew 500 ppm
I20260812 11:12:03.191943 8381 webserver.cc:460] Webserver started at http://127.23.97.254:34963/ using document root <none> and password file <none>
I20260812 11:12:03.192587 8381 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:03.192652 8381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:03.192900 8381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:03.194880 8381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "541bb0f51b9142ddae69ab30d8f6a0aa"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "e5a155472bb7d3eb4ff7afe595ceae3a"
server_key_iv: "33cc74e7ac2748374dab44841f928029"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.195439 8381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "541bb0f51b9142ddae69ab30d8f6a0aa"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "e5a155472bb7d3eb4ff7afe595ceae3a"
server_key_iv: "33cc74e7ac2748374dab44841f928029"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.199039 8381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:12:03.201486 8407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.202674 8381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:12:03.202805 8381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "541bb0f51b9142ddae69ab30d8f6a0aa"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "e5a155472bb7d3eb4ff7afe595ceae3a"
server_key_iv: "33cc74e7ac2748374dab44841f928029"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.202909 8381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:03.230652 8381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:03.231357 8381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:03.231571 8381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:03.238637 8381 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:35323
I20260812 11:12:03.238673 8517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:35323 every 8 connection(s)
I20260812 11:12:03.239619 8381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:12:03.242858 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:03.248270 8518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa: Bootstrap starting.
I20260812 11:12:03.249511 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 8381
I20260812 11:12:03.249639 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:12:03.249969 23943 external_mini_cluster.cc:1428] Setting key cf8b7f6d019df9c165dd85cfbfe48410
I20260812 11:12:03.251216 8518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:03.251962 8518 log.cc:824] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:03.254102 8518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa: No bootstrap required, opened a new log
I20260812 11:12:03.258318 8518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } }
I20260812 11:12:03.258620 8518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:03.258687 8518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541bb0f51b9142ddae69ab30d8f6a0aa, State: Initialized, Role: FOLLOWER
I20260812 11:12:03.259156 8518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } }
I20260812 11:12:03.259287 8518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:12:03.259374 8518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:12:03.259470 8518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:03.259675 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.251135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.23.97.254:35323)
0812 11:12:03.251784 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:03.251806 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:03.251840 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:12:03.252638 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:03.252673 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:03.253075 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:03.253356 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:03.253716 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.254754 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.256034 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.256594 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.256721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.257656 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:03.257662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:03.257674 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:03.258049 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:03.258132 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:03.258154 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:03.258232 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:03.258408 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 11:12:03.258729 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":174,"threads_started":1}
I20260812 11:12:03.260458 8518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } }
I20260812 11:12:03.260833 8518 leader_election.cc:302] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541bb0f51b9142ddae69ab30d8f6a0aa; no voters:
I20260812 11:12:03.261157 8518 leader_election.cc:288] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:12:03.261377 8528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:03.261651 8528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [term 1 LEADER]: Becoming Leader. State: Replica: 541bb0f51b9142ddae69ab30d8f6a0aa, State: Running, Role: LEADER
I20260812 11:12:03.262316 8528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } }
I20260812 11:12:03.262391 8518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:12:03.263458 8528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541bb0f51b9142ddae69ab30d8f6a0aa. Latest consensus state: current_term: 1 leader_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } } }
I20260812 11:12:03.263590 8528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:03.263877 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541bb0f51b9142ddae69ab30d8f6a0aa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35323 } } }
I20260812 11:12:03.263970 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:03.264022 8538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:12:03.267236 8538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:12:03.272815 8538 catalog_manager.cc:1347] Generated new cluster ID: 464263a406e5433fb10b6a15d0efe96f
I20260812 11:12:03.272881 8538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:12:03.281298 8538 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:12:03.282455 8538 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:12:03.292217 8538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541bb0f51b9142ddae69ab30d8f6a0aa: Generated new TSK 0
I20260812 11:12:03.292948 8538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:12:03.301100 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--builtin_ntp_servers=127.23.97.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 11:12:03.417148 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:03.417449 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:03.417521 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:03.421605 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:03.421711 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:03.421802 8555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:12:03.426684 8555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:03.427829 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:03.429287 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:12:03.439113 8555 server_base.cc:1034] running on GCE node
W20260812 11:12:03.439066 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.439064 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.439078 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:03.439774 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:03.440490 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:03.441670 8555 hybrid_clock.cc:648] HybridClock initialized: now 1786533123441661 us; error 38 us; skew 500 ppm
I20260812 11:12:03.443638 8555 webserver.cc:460] Webserver started at http://127.23.97.193:33729/ using document root <none> and password file <none>
I20260812 11:12:03.444259 8555 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:03.444360 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:03.444605 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:03.446731 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "748de25bcf384e4ca905ee475787a498"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "748b664a0c1d3855f8b9df6d44b29ce6"
server_key_iv: "84127126af102b8760e79971576c1fed"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.447314 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "748de25bcf384e4ca905ee475787a498"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "748b664a0c1d3855f8b9df6d44b29ce6"
server_key_iv: "84127126af102b8760e79971576c1fed"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.451160 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:12:03.453847 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.455153 8555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:12:03.455328 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "748de25bcf384e4ca905ee475787a498"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "748b664a0c1d3855f8b9df6d44b29ce6"
server_key_iv: "84127126af102b8760e79971576c1fed"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.455458 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:03.470255 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:03.470906 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:03.471110 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:03.471670 8555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:03.472669 8555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:03.472744 8555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.472816 8555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:03.472860 8555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.481940 8555 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:42443
I20260812 11:12:03.481994 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:42443 every 8 connection(s)
I20260812 11:12:03.482879 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:12:03.488669 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 8555
I20260812 11:12:03.488794 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:12:03.489090 23943 external_mini_cluster.cc:1428] Setting key 5ea14c602637127fd293f5476e98b6cc
I20260812 11:12:03.491261 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--builtin_ntp_servers=127.23.97.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:12:03.493425 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.484784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:50633 (local address 127.23.97.254:35323)
0812 11:12:03.484959 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:03.484964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:03.486052 (+ 1088us) server_negotiation.cc:407] Connection header received
0812 11:12:03.487407 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:03.487409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:03.487457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:03.487535 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:03.488512 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.489295 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.490859 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.491306 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.491446 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.492874 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:03.492878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:03.492881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:03.492895 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:03.492925 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:03.492934 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:03.493026 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:03.493207 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 11:12:03.493272 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:12:03.494385 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.485208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35323 (local address 127.23.97.193:50633)
0812 11:12:03.485894 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:03.485935 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:03.487197 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:03.487716 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:03.487722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:03.487803 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:03.488293 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.488306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.489476 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.489481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.490705 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.490718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.491641 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.491648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.491863 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.492587 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:03.492589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:03.492632 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:03.492700 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:03.493030 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:03.493044 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:03.493052 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:12:03.493206 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 11:12:03.493505 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260812 11:12:03.495645 8816 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35323
I20260812 11:12:03.495900 8816 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:03.496582 8816 heartbeater.cc:511] Master 127.23.97.254:35323 requested a full tablet report, sending...
I20260812 11:12:03.498227 8450 ts_manager.cc:194] Registered new tserver with Master: 748de25bcf384e4ca905ee475787a498 (127.23.97.193:42443)
I20260812 11:12:03.499488 8450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.193:50633
W20260812 11:12:03.605142 8819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:03.605377 8819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:03.605412 8819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:03.609537 8819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:03.609588 8819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:03.609665 8819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:12:03.614234 8819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:03.615286 8819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:03.616667 8819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:03.626780 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.626796 8837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.626788 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:03.627251 8819 server_base.cc:1034] running on GCE node
I20260812 11:12:03.627712 8819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:03.628449 8819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:03.629647 8819 hybrid_clock.cc:648] HybridClock initialized: now 1786533123629625 us; error 32 us; skew 500 ppm
I20260812 11:12:03.631693 8819 webserver.cc:460] Webserver started at http://127.23.97.194:39945/ using document root <none> and password file <none>
I20260812 11:12:03.632323 8819 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:03.632479 8819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:03.632750 8819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:03.635043 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "a4363c611a024db5858e7019a32388b2"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "b7dd5a64ca68b0c4c300c3446f79d9e7"
server_key_iv: "be7d6e9c7896891d3d69a52f6c6cc8a1"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.635670 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "a4363c611a024db5858e7019a32388b2"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "b7dd5a64ca68b0c4c300c3446f79d9e7"
server_key_iv: "be7d6e9c7896891d3d69a52f6c6cc8a1"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.639742 8819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 11:12:03.642493 8847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.643812 8819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:12:03.643959 8819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "a4363c611a024db5858e7019a32388b2"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "b7dd5a64ca68b0c4c300c3446f79d9e7"
server_key_iv: "be7d6e9c7896891d3d69a52f6c6cc8a1"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.644071 8819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:03.656636 8819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:03.657297 8819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:03.657498 8819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:03.658138 8819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:03.659255 8819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:03.659307 8819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.659379 8819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:03.659420 8819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.668969 8819 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:36991
I20260812 11:12:03.669041 9063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:36991 every 8 connection(s)
I20260812 11:12:03.669961 8819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:12:03.678643 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 8819
I20260812 11:12:03.678792 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:12:03.679190 23943 external_mini_cluster.cc:1428] Setting key 9df7704ee0429aeee92ae96e4553f3cd
I20260812 11:12:03.679422 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.671798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:34229 (local address 127.23.97.254:35323)
0812 11:12:03.671954 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:03.671958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:03.673001 (+ 1043us) server_negotiation.cc:407] Connection header received
0812 11:12:03.674435 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:03.674439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:03.674500 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:03.674610 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:03.675503 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.676052 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.677292 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.677518 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.677625 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.678857 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:03.678861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:03.678866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:03.678884 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:03.678914 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:03.678923 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:03.678989 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:03.679191 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 11:12:03.679263 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:12:03.680305 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.672154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35323 (local address 127.23.97.194:34229)
0812 11:12:03.672855 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:03.672895 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:03.674215 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:03.674786 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:03.674792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:03.674872 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:03.675349 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.675363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.676220 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.676223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.677142 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.677152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.677673 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.677677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.677840 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.678508 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:03.678511 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:03.678568 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:03.678649 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:03.679018 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:03.679029 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:03.679036 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:12:03.679178 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 11:12:03.679461 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":164,"threads_started":1}
I20260812 11:12:03.681305 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--builtin_ntp_servers=127.23.97.212:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:12:03.681432 9066 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35323
I20260812 11:12:03.681680 9066 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:03.682166 9066 heartbeater.cc:511] Master 127.23.97.254:35323 requested a full tablet report, sending...
I20260812 11:12:03.683214 8450 ts_manager.cc:194] Registered new tserver with Master: a4363c611a024db5858e7019a32388b2 (127.23.97.194:36991)
I20260812 11:12:03.684023 8450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.194:34229
W20260812 11:12:03.794850 9073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:03.795096 9073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:03.795131 9073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:03.799105 9073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:03.799157 9073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:03.799214 9073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:12:03.803906 9073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:03.805001 9073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:03.806335 9073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:03.815920 9086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:03.816102 9073 server_base.cc:1034] running on GCE node
W20260812 11:12:03.815974 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:03.815977 9085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:03.816645 9073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:03.817276 9073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:03.818439 9073 hybrid_clock.cc:648] HybridClock initialized: now 1786533123818425 us; error 27 us; skew 500 ppm
I20260812 11:12:03.820227 9073 webserver.cc:460] Webserver started at http://127.23.97.195:38289/ using document root <none> and password file <none>
I20260812 11:12:03.820848 9073 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:03.820912 9073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:03.821180 9073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:03.823297 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "eda71ef0e6974e6aa9cb9a9c0dc85079"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "c8ae0711a3585d663ab52f6e43d591b7"
server_key_iv: "f6061213a739ea76e25696fd43781ab5"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.823879 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "eda71ef0e6974e6aa9cb9a9c0dc85079"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "c8ae0711a3585d663ab52f6e43d591b7"
server_key_iv: "f6061213a739ea76e25696fd43781ab5"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.827584 9073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:12:03.830156 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.831318 9073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:12:03.831465 9073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "eda71ef0e6974e6aa9cb9a9c0dc85079"
format_stamp: "Formatted at 2026-08-12 11:12:03 on dist-test-slave-zpvx"
server_key: "c8ae0711a3585d663ab52f6e43d591b7"
server_key_iv: "f6061213a739ea76e25696fd43781ab5"
server_key_version: "encryptionkey@0"
I20260812 11:12:03.831579 9073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:03.865289 9073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:03.866065 9073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:03.866289 9073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:03.866909 9073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:03.867906 9073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:03.867981 9073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.868049 9073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:03.868093 9073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:03.877085 9073 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:43279
I20260812 11:12:03.877112 9302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:43279 every 8 connection(s)
I20260812 11:12:03.878019 9073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:12:03.878707 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 9073
I20260812 11:12:03.878849 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:12:03.879132 23943 external_mini_cluster.cc:1428] Setting key e2842d3b8972774c109f054469ffbb9d
I20260812 11:12:03.887596 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.880072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:53683 (local address 127.23.97.254:35323)
0812 11:12:03.880222 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:03.880229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:03.881873 (+ 1644us) server_negotiation.cc:407] Connection header received
0812 11:12:03.883197 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:03.883200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:03.883244 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:03.883313 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:03.884138 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.884850 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.885938 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.886149 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.886227 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.887200 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:03.887203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:03.887205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:03.887217 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:03.887234 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:03.887241 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:03.887310 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:03.887443 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 11:12:03.887489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:12:03.888463 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:03.880510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35323 (local address 127.23.97.195:53683)
0812 11:12:03.881732 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:03.881773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:03.883017 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:03.883441 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:03.883447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:03.883526 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:03.883998 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.884011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.884959 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.884962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.885834 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:03.885843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:03.886279 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:03.886282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:03.886419 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:03.886966 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:03.886968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:03.887006 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:03.887067 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:03.887323 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:03.887330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:03.887334 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:12:03.887423 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:12:03.887626 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":125,"threads_started":1}
I20260812 11:12:03.889549 9305 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35323
I20260812 11:12:03.889780 9305 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:03.890271 9305 heartbeater.cc:511] Master 127.23.97.254:35323 requested a full tablet report, sending...
I20260812 11:12:03.891287 8450 ts_manager.cc:194] Registered new tserver with Master: eda71ef0e6974e6aa9cb9a9c0dc85079 (127.23.97.195:43279)
I20260812 11:12:03.891932 8450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.195:53683
I20260812 11:12:03.893335 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:12:03.896507 8523 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:03.895471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.23.97.254:35323)
0812 11:12:03.895635 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:03.895640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:03.895851 (+ 211us) server_negotiation.cc:407] Connection header received
0812 11:12:03.895914 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:03.895917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:03.895957 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:03.896030 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:03.896313 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39580: BlockingRecv error: recv got EOF from 127.0.0.1:39580 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:12:03.896852 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 8555
I20260812 11:12:03.918623 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 8819
I20260812 11:12:03.938547 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 9073
I20260812 11:12:03.948083 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 8381
2026-08-12T11:12:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (927 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T11:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:12:03Z Disabled control of system clock
I20260812 11:12:03.982111 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:37241
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:12:04.093662 9326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:04.093907 9326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:04.093945 9326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:04.097903 9326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:12:04.097956 9326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:04.097971 9326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:04.097988 9326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:12:04.098003 9326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:12:04.102553 9326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:37241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:04.103549 9326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:04.104960 9326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:04.113750 9333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.113808 9335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.113741 9342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:04.113806 9326 server_base.cc:1034] running on GCE node
I20260812 11:12:04.114621 9326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:04.115777 9326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:04.116946 9326 hybrid_clock.cc:648] HybridClock initialized: now 1786533124116930 us; error 28 us; skew 500 ppm
I20260812 11:12:04.118870 9326 webserver.cc:460] Webserver started at http://127.23.97.254:33069/ using document root <none> and password file <none>
I20260812 11:12:04.119482 9326 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:04.119546 9326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:04.119808 9326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:04.121950 9326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "e9b8199b76974a29b103cd71de2244d0"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "1899848faf640b57e886d87584f855c6"
server_key_iv: "b5c207385f1cbe4f3a3f3503ef1a603f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.122534 9326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "e9b8199b76974a29b103cd71de2244d0"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "1899848faf640b57e886d87584f855c6"
server_key_iv: "b5c207385f1cbe4f3a3f3503ef1a603f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.126255 9326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:12:04.128683 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.129894 9326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:12:04.130039 9326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e9b8199b76974a29b103cd71de2244d0"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "1899848faf640b57e886d87584f855c6"
server_key_iv: "b5c207385f1cbe4f3a3f3503ef1a603f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.130141 9326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:04.135304 9326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:04.135921 9326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:04.136116 9326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:04.143155 9326 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:37241
I20260812 11:12:04.143162 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:37241 every 8 connection(s)
I20260812 11:12:04.144125 9326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:12:04.147423 9458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:04.147883 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 9326
I20260812 11:12:04.148012 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:12:04.148342 23943 external_mini_cluster.cc:1428] Setting key 32b3aea5854e217dc2acf25faed27fec
I20260812 11:12:04.153918 9458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0: Bootstrap starting.
I20260812 11:12:04.154093 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.149816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.23.97.254:37241)
0812 11:12:04.150491 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:04.150507 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:04.150542 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:12:04.151204 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:04.151222 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:04.151573 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:04.151923 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:04.152780 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:04.152784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:04.152796 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:04.152808 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:04.152865 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:04.152881 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:04.153036 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:04.153111 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 11:12:04.153358 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260812 11:12:04.156188 9458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:04.156996 9458 log.cc:824] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:04.159176 9458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0: No bootstrap required, opened a new log
I20260812 11:12:04.162811 9458 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } }
I20260812 11:12:04.163039 9458 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:04.163074 9458 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b8199b76974a29b103cd71de2244d0, State: Initialized, Role: FOLLOWER
I20260812 11:12:04.163446 9458 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } }
I20260812 11:12:04.163605 9458 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:12:04.163678 9458 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:12:04.163787 9458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:04.164915 9458 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } }
I20260812 11:12:04.165261 9458 leader_election.cc:302] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b8199b76974a29b103cd71de2244d0; no voters:
I20260812 11:12:04.165571 9458 leader_election.cc:288] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:12:04.165879 9463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:04.166133 9463 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [term 1 LEADER]: Becoming Leader. State: Replica: e9b8199b76974a29b103cd71de2244d0, State: Running, Role: LEADER
I20260812 11:12:04.166445 9463 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } }
I20260812 11:12:04.166853 9458 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:12:04.168200 9467 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b8199b76974a29b103cd71de2244d0. Latest consensus state: current_term: 1 leader_uuid: "e9b8199b76974a29b103cd71de2244d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } } }
I20260812 11:12:04.168211 9465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b8199b76974a29b103cd71de2244d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b8199b76974a29b103cd71de2244d0" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37241 } } }
I20260812 11:12:04.168355 9465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:04.168344 9467 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:04.169750 9484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:12:04.172588 9484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:12:04.178272 9484 catalog_manager.cc:1347] Generated new cluster ID: 296b94fe39af497f8bc68c180bb70960
I20260812 11:12:04.178352 9484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:12:04.189280 9484 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:12:04.189605 9484 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:12:04.194535 9484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b8199b76974a29b103cd71de2244d0: Generated new TSK 0
I20260812 11:12:04.195240 9484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:12:04.204514 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--builtin_ntp_servers=127.23.97.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:12:04.320031 9509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:04.320269 9509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:04.320343 9509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:04.324337 9509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:04.324442 9509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:04.324502 9509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:12:04.329119 9509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:04.330014 9509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:04.331353 9509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:12:04.340981 9509 server_base.cc:1034] running on GCE node
W20260812 11:12:04.340885 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.341023 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.340912 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:04.341581 9509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:04.342211 9509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:04.343401 9509 hybrid_clock.cc:648] HybridClock initialized: now 1786533124343387 us; error 26 us; skew 500 ppm
I20260812 11:12:04.345257 9509 webserver.cc:460] Webserver started at http://127.23.97.193:42431/ using document root <none> and password file <none>
I20260812 11:12:04.345850 9509 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:04.345944 9509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:04.346179 9509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:04.348253 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "27a5657b3e424fd081a7caa90d44d397"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "ec3e00e580d8a8e6a57d49116484129f"
server_key_iv: "a827e470a8b74fcf58dfb6afb033cdf8"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.348882 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "27a5657b3e424fd081a7caa90d44d397"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "ec3e00e580d8a8e6a57d49116484129f"
server_key_iv: "a827e470a8b74fcf58dfb6afb033cdf8"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.352541 9509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:12:04.354945 9536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.356084 9509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:12:04.356252 9509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "27a5657b3e424fd081a7caa90d44d397"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "ec3e00e580d8a8e6a57d49116484129f"
server_key_iv: "a827e470a8b74fcf58dfb6afb033cdf8"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.356369 9509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:04.360910 9509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:04.361567 9509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:04.361752 9509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:04.362285 9509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:04.363168 9509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:04.363247 9509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.363319 9509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:04.363366 9509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.372915 9509 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33541
I20260812 11:12:04.372989 9743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33541 every 8 connection(s)
I20260812 11:12:04.373797 9509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:12:04.379844 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.375590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:59595 (local address 127.23.97.254:37241)
0812 11:12:04.375733 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:04.375743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:04.376761 (+ 1018us) server_negotiation.cc:407] Connection header received
0812 11:12:04.378079 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:04.378083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:04.378130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:04.378232 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:04.379356 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:04.379360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:04.379362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:04.379376 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:04.379397 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:04.379413 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:04.379513 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:04.379662 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 11:12:04.379708 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:12:04.380707 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.375905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:37241 (local address 127.23.97.193:59595)
0812 11:12:04.376604 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:04.376643 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:04.377861 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:04.378393 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:04.378400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:04.378483 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:04.379089 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:04.379090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:04.379129 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:04.379197 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:04.379498 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:04.379507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:04.379513 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:12:04.379712 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 11:12:04.379941 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260812 11:12:04.381765 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 9509
I20260812 11:12:04.381886 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:12:04.381950 9745 heartbeater.cc:347] Connected to a master server at 127.23.97.254:37241
I20260812 11:12:04.382184 9745 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:04.382227 23943 external_mini_cluster.cc:1428] Setting key c6142acfaaf282cc8f57633b4eae38b5
I20260812 11:12:04.382648 9745 heartbeater.cc:511] Master 127.23.97.254:37241 requested a full tablet report, sending...
I20260812 11:12:04.384230 9375 ts_manager.cc:194] Registered new tserver with Master: 27a5657b3e424fd081a7caa90d44d397 (127.23.97.193:33541)
I20260812 11:12:04.384326 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--builtin_ntp_servers=127.23.97.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:12:04.497089 9754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:04.497337 9754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:04.497376 9754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:04.501410 9754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:04.501459 9754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:04.501545 9754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:12:04.506222 9754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:04.507212 9754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:04.508656 9754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:04.518322 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.518324 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:04.518365 9754 server_base.cc:1034] running on GCE node
W20260812 11:12:04.518323 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:04.518906 9754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:04.519552 9754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:04.520758 9754 hybrid_clock.cc:648] HybridClock initialized: now 1786533124520739 us; error 29 us; skew 500 ppm
I20260812 11:12:04.522655 9754 webserver.cc:460] Webserver started at http://127.23.97.194:45111/ using document root <none> and password file <none>
I20260812 11:12:04.523228 9754 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:04.523321 9754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:04.523553 9754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:04.525641 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "ed0ee236db004c31ae8c0faa8dfc5bcd"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "18bf2b4a039a0086322ca054cc45e0e4"
server_key_iv: "fd1debd69dde9e3c22506b19f29f209f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.526211 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0ee236db004c31ae8c0faa8dfc5bcd"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "18bf2b4a039a0086322ca054cc45e0e4"
server_key_iv: "fd1debd69dde9e3c22506b19f29f209f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.529793 9754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:12:04.532243 9776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.533417 9754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:12:04.533556 9754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "ed0ee236db004c31ae8c0faa8dfc5bcd"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "18bf2b4a039a0086322ca054cc45e0e4"
server_key_iv: "fd1debd69dde9e3c22506b19f29f209f"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.533689 9754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:04.538371 9754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:04.538954 9754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:04.539146 9754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:04.539686 9754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:04.540629 9754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:04.540679 9754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.540743 9754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:04.540781 9754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.549752 9754 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:39061
I20260812 11:12:04.549846 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:39061 every 8 connection(s)
I20260812 11:12:04.550640 9754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:12:04.551339 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 9754
I20260812 11:12:04.551470 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:12:04.551805 23943 external_mini_cluster.cc:1428] Setting key 3295016029b02aac18068a7ee66fcace
I20260812 11:12:04.554101 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--builtin_ntp_servers=127.23.97.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 11:12:04.558182 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.552716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:39973 (local address 127.23.97.254:37241)
0812 11:12:04.552856 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:04.552860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:04.554114 (+ 1254us) server_negotiation.cc:407] Connection header received
0812 11:12:04.556100 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:04.556105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:04.556157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:04.556216 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:04.557552 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:04.557555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:04.557557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:04.557594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:04.557621 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:04.557630 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:04.557708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:04.557946 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 11:12:04.557997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:12:04.559669 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.553132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:37241 (local address 127.23.97.194:39973)
0812 11:12:04.553936 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:04.553993 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:04.555839 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:04.556599 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:04.556605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:04.556687 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:04.557271 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:04.557273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:04.557323 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:04.557386 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:04.557724 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:04.557736 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:04.557745 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:12:04.558403 (+ 658us) client_negotiation.cc:240] Negotiation successful
0812 11:12:04.558638 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
I20260812 11:12:04.560791 10001 heartbeater.cc:347] Connected to a master server at 127.23.97.254:37241
I20260812 11:12:04.561038 10001 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:04.561535 10001 heartbeater.cc:511] Master 127.23.97.254:37241 requested a full tablet report, sending...
I20260812 11:12:04.562718 9375 ts_manager.cc:194] Registered new tserver with Master: ed0ee236db004c31ae8c0faa8dfc5bcd (127.23.97.194:39061)
W20260812 11:12:04.665313 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:04.665548 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:04.665588 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:04.669839 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:04.669888 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:04.669970 10005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:12:04.674607 10005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:04.675577 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:04.676898 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:04.686270 10023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.686264 10015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:04.686673 10025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:04.686870 10005 server_base.cc:1034] running on GCE node
I20260812 11:12:04.687376 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:04.688154 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:04.689356 10005 hybrid_clock.cc:648] HybridClock initialized: now 1786533124689331 us; error 56 us; skew 500 ppm
I20260812 11:12:04.691651 10005 webserver.cc:460] Webserver started at http://127.23.97.195:40367/ using document root <none> and password file <none>
I20260812 11:12:04.692217 10005 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:04.692328 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:04.692602 10005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:04.695365 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "ae72492d80704cd89a09773f0d5ea6b1"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "d8a2365c94d9dfc9dcc18515d7269d56"
server_key_iv: "2acfcb85facd5d4aa7828cfbb29a0788"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.695869 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "ae72492d80704cd89a09773f0d5ea6b1"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "d8a2365c94d9dfc9dcc18515d7269d56"
server_key_iv: "2acfcb85facd5d4aa7828cfbb29a0788"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.699595 10005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:12:04.702446 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.703696 10005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:12:04.703794 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "ae72492d80704cd89a09773f0d5ea6b1"
format_stamp: "Formatted at 2026-08-12 11:12:04 on dist-test-slave-zpvx"
server_key: "d8a2365c94d9dfc9dcc18515d7269d56"
server_key_iv: "2acfcb85facd5d4aa7828cfbb29a0788"
server_key_version: "encryptionkey@0"
I20260812 11:12:04.703943 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:04.708494 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:04.709105 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:04.709292 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:04.709837 10005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:04.710824 10005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:04.710870 10005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.710934 10005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:04.710973 10005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:04.720436 10005 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:34183
I20260812 11:12:04.720466 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:34183 every 8 connection(s)
I20260812 11:12:04.721271 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:12:04.721800 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 10005
I20260812 11:12:04.721921 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:12:04.722237 23943 external_mini_cluster.cc:1428] Setting key f2881c76bef3f5e3f6ebaf3ffd0cb77c
I20260812 11:12:04.727540 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.723203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:53079 (local address 127.23.97.254:37241)
0812 11:12:04.723346 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:04.723350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:04.724634 (+ 1284us) server_negotiation.cc:407] Connection header received
0812 11:12:04.725946 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:04.725949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:04.725988 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:04.726082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:04.727189 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:04.727192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:04.727194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:12:04.727205 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:04.727224 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:12:04.727232 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:04.727307 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:04.727398 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 11:12:04.727433 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:12:04.728356 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:04.723680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:37241 (local address 127.23.97.195:53079)
0812 11:12:04.724458 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:04.724508 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:04.725740 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:04.726223 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:04.726229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:04.726333 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:04.726935 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:12:04.726936 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:04.726975 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:12:04.727042 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:04.727294 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:04.727298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:04.727302 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:12:04.727417 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 11:12:04.727621 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260812 11:12:04.729530 10251 heartbeater.cc:347] Connected to a master server at 127.23.97.254:37241
I20260812 11:12:04.729727 10251 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:04.730069 10251 heartbeater.cc:511] Master 127.23.97.254:37241 requested a full tablet report, sending...
I20260812 11:12:04.731011 9375 ts_manager.cc:194] Registered new tserver with Master: ae72492d80704cd89a09773f0d5ea6b1 (127.23.97.195:34183)
I20260812 11:12:04.736299 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:12:04.739107 9461 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:04.738361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.23.97.254:37241)
0812 11:12:04.738478 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:04.738482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:04.738632 (+ 150us) server_negotiation.cc:407] Connection header received
0812 11:12:04.738722 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:04.738725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:04.738755 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:04.738825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:04.739013 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36846: BlockingRecv error: recv got EOF from 127.0.0.1:36846 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:12:04.739305 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 9509
I20260812 11:12:04.751613 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 9754
I20260812 11:12:04.781080 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 10005
I20260812 11:12:04.791599 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 9326
2026-08-12T11:12:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (854 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:12:04 dist-test-slave-zpvx krb5kdc[10270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:12:04 dist-test-slave-zpvx krb5kdc[10270](info): set up 2 sockets
Aug 12 11:12:04 dist-test-slave-zpvx krb5kdc[10270](info): commencing operation
krb5kdc: starting...
W20260812 11:12:06.129313 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:12:06.255208 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39137
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:06.363528 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:06.363790 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:06.363859 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:06.367913 10299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:12:06.367990 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:06.368026 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:06.368081 10299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:12:06.368121 10299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:12:06.373389 10299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:39137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:06.374356 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:06.375315 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:06.383589 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.383605 10308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.383591 10311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:06.383811 10299 server_base.cc:1034] running on GCE node
I20260812 11:12:06.384300 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:06.385382 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:06.386543 10299 hybrid_clock.cc:648] HybridClock initialized: now 1786533126386524 us; error 32 us; skew 500 ppm
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:06.389156 10299 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:12:06.390014 10299 webserver.cc:460] Webserver started at http://127.23.97.254:43951/ using document root <none> and password file <none>
I20260812 11:12:06.390568 10299 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:06.390656 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:06.390882 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:06.392886 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "e540c569c8214ac1af1b26a6175b525d"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "35f869d99d98e7cb0241f9606505d2b2"
server_key_iv: "41463b8a0b234ca8b57e6f1919812e10"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.393446 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "e540c569c8214ac1af1b26a6175b525d"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "35f869d99d98e7cb0241f9606505d2b2"
server_key_iv: "41463b8a0b234ca8b57e6f1919812e10"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.397027 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:12:06.399361 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.400599 10299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:12:06.400732 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e540c569c8214ac1af1b26a6175b525d"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "35f869d99d98e7cb0241f9606505d2b2"
server_key_iv: "41463b8a0b234ca8b57e6f1919812e10"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.400839 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:06.421677 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:06.424468 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:06.424674 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:06.431586 10299 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:39137
I20260812 11:12:06.431921 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:39137 every 8 connection(s)
I20260812 11:12:06.432564 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:12:06.435307 10433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:06.440719 10433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: Bootstrap starting.
I20260812 11:12:06.441027 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 10299
I20260812 11:12:06.441169 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:12:06.441460 23943 external_mini_cluster.cc:1428] Setting key 1fd243f3b7b2cde1286bd34a4f2ff898
I20260812 11:12:06.443423 10433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:06.444150 10433 log.cc:824] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:06.446215 10433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: No bootstrap required, opened a new log
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:06.450706 10433 raft_consensus.cc:348] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } }
I20260812 11:12:06.450985 10433 raft_consensus.cc:374] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:06.451054 10433 raft_consensus.cc:729] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e540c569c8214ac1af1b26a6175b525d, State: Initialized, Role: FOLLOWER
I20260812 11:12:06.451575 10433 consensus_queue.cc:260] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } }
I20260812 11:12:06.451763 10433 raft_consensus.cc:388] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:12:06.451846 10433 raft_consensus.cc:482] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:12:06.451933 10433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:06.453079 10433 raft_consensus.cc:504] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } }
I20260812 11:12:06.453415 10433 leader_election.cc:302] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e540c569c8214ac1af1b26a6175b525d; no voters:
I20260812 11:12:06.453747 10433 leader_election.cc:288] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:12:06.453850 10446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:06.454099 10446 raft_consensus.cc:686] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [term 1 LEADER]: Becoming Leader. State: Replica: e540c569c8214ac1af1b26a6175b525d, State: Running, Role: LEADER
I20260812 11:12:06.454397 10446 consensus_queue.cc:237] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } }
I20260812 11:12:06.455076 10433 sys_catalog.cc:564] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:12:06.455931 10448 sys_catalog.cc:455] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e540c569c8214ac1af1b26a6175b525d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } } }
I20260812 11:12:06.456049 10448 sys_catalog.cc:458] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:06.456475 10452 sys_catalog.cc:455] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e540c569c8214ac1af1b26a6175b525d. Latest consensus state: current_term: 1 leader_uuid: "e540c569c8214ac1af1b26a6175b525d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e540c569c8214ac1af1b26a6175b525d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39137 } } }
I20260812 11:12:06.456569 10452 sys_catalog.cc:458] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:06.457473 10470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:12:06.457913 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:06.442748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.23.97.254:39137)
0812 11:12:06.443142 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:06.443157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:06.443192 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:12:06.443854 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:06.443873 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:06.444182 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:06.444575 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:06.445390 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.446384 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.447518 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.447966 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.448081 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:06.450935 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:06.450944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:06.450963 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:06.450995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:06.453088 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.453594 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.453600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.453606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.453783 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.454156 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.454159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.454161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.454642 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:06.454834 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:06.455340 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 11:12:06.456894 (+ 1554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":131,"threads_started":1}
I20260812 11:12:06.460680 10470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:12:06.466218 10470 catalog_manager.cc:1347] Generated new cluster ID: 2f5c5bdceef74b578d21e2c49514260d
I20260812 11:12:06.466295 10470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:12:06.473179 10470 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:12:06.474488 10470 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:12:06.481355 10470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: Generated new TSK 0
I20260812 11:12:06.482081 10470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:12:06.512064 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:06.624048 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:06.624321 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:06.624447 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:06.628543 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:06.628624 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:06.628715 10499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:12:06.633829 10499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:06.634832 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:06.635841 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:06.645216 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.645221 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.645263 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:06.645526 10499 server_base.cc:1034] running on GCE node
I20260812 11:12:06.645895 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:06.646499 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:06.647686 10499 hybrid_clock.cc:648] HybridClock initialized: now 1786533126647674 us; error 35 us; skew 500 ppm
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:06.650472 10499 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:12:06.651268 10499 webserver.cc:460] Webserver started at http://127.23.97.193:41477/ using document root <none> and password file <none>
I20260812 11:12:06.651787 10499 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:06.651844 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:06.652005 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:06.654024 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "ba018a18b1c848e0a2a900889a98c464"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "c40418c6f698bc2cf383bf953804ccd5"
server_key_iv: "83f8562afe3282ac0a3f9ed41bde0b64"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.654544 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "ba018a18b1c848e0a2a900889a98c464"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "c40418c6f698bc2cf383bf953804ccd5"
server_key_iv: "83f8562afe3282ac0a3f9ed41bde0b64"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.658456 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:12:06.660849 10531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.662058 10499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:12:06.662199 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "ba018a18b1c848e0a2a900889a98c464"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "c40418c6f698bc2cf383bf953804ccd5"
server_key_iv: "83f8562afe3282ac0a3f9ed41bde0b64"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.662308 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:06.671882 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:06.674757 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:06.674957 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:06.675549 10499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:06.676463 10499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:06.676530 10499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.676597 10499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:06.676643 10499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.685840 10499 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:39455
I20260812 11:12:06.686049 10741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:39455 every 8 connection(s)
I20260812 11:12:06.686789 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:12:06.688558 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 10499
I20260812 11:12:06.688699 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:12:06.689059 23943 external_mini_cluster.cc:1428] Setting key ee2e32ecdcb29606d9a995bf122ee6ff
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:06.710016 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:06.689201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:55463 (local address 127.23.97.254:39137)
0812 11:12:06.689361 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:06.689366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:06.691049 (+ 1683us) server_negotiation.cc:407] Connection header received
0812 11:12:06.693045 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:06.693051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:06.693117 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:06.693223 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:06.694928 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.695467 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.697444 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.697701 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.697784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:06.705914 (+ 8130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:06.705921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:06.705925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:06.705961 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:06.707554 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.708248 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.708251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.708252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.708328 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.708779 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.708781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.708783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.708958 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:06.709032 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:06.709785 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 11:12:06.709891 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:12:06.711258 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:06.689885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39137 (local address 127.23.97.193:55463)
0812 11:12:06.690887 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:06.690950 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:06.692810 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:06.693464 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:06.693476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:06.694027 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:06.694757 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:06.694782 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.695610 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.695617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:06.697309 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:06.697326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.697871 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.697879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:06.698145 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:06.703803 (+ 5658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:06.703821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:06.705731 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:06.707684 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:06.707694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:06.707715 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:06.708117 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:06.708472 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:06.708475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:06.708478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:06.708676 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:06.709092 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:06.709102 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:06.709481 (+ 379us) client_negotiation.cc:769] Sending connection context
0812 11:12:06.709718 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 11:12:06.710113 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":145,"threads_started":1}
I20260812 11:12:06.712663 10742 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39137
I20260812 11:12:06.712895 10742 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:06.713413 10742 heartbeater.cc:511] Master 127.23.97.254:39137 requested a full tablet report, sending...
I20260812 11:12:06.714941 10359 ts_manager.cc:194] Registered new tserver with Master: ba018a18b1c848e0a2a900889a98c464 (127.23.97.193:39455)
I20260812 11:12:06.716064 10359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:55463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:12:06.721326 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:06.834249 10750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:06.834511 10750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:06.834554 10750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:06.838557 10750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:06.838613 10750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:06.838675 10750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:12:06.843837 10750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:06.844859 10750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:06.845870 10750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:06.855464 10764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.855509 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:06.855513 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:06.855702 10750 server_base.cc:1034] running on GCE node
I20260812 11:12:06.856185 10750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:06.856832 10750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:06.858062 10750 hybrid_clock.cc:648] HybridClock initialized: now 1786533126858045 us; error 37 us; skew 500 ppm
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:06.860812 10750 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:12:06.861654 10750 webserver.cc:460] Webserver started at http://127.23.97.194:43813/ using document root <none> and password file <none>
I20260812 11:12:06.862226 10750 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:06.862320 10750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:06.862558 10750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:06.864735 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "31c688ad0af442209215aee0f9fbc7a9"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "dcded0375505e585a3716b380d93b5ef"
server_key_iv: "2794d59f7e351baf1395961c2cfde591"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.865345 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "31c688ad0af442209215aee0f9fbc7a9"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "dcded0375505e585a3716b380d93b5ef"
server_key_iv: "2794d59f7e351baf1395961c2cfde591"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.869364 10750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:12:06.871805 10790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.873057 10750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:12:06.873194 10750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "31c688ad0af442209215aee0f9fbc7a9"
format_stamp: "Formatted at 2026-08-12 11:12:06 on dist-test-slave-zpvx"
server_key: "dcded0375505e585a3716b380d93b5ef"
server_key_iv: "2794d59f7e351baf1395961c2cfde591"
server_key_version: "encryptionkey@0"
I20260812 11:12:06.873322 10750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:06.890621 10750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:06.893342 10750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:06.893551 10750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:06.894110 10750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:06.894907 10750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:06.894976 10750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.895046 10750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:06.895092 10750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:06.904027 10750 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:33035
I20260812 11:12:06.904071 11004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:33035 every 8 connection(s)
I20260812 11:12:06.905027 10750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:12:06.908046 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 10750
I20260812 11:12:06.908191 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:12:06.908591 23943 external_mini_cluster.cc:1428] Setting key f6f4fa1d7f2fcfaf895b411227b99fc5
Aug 12 11:12:06 dist-test-slave-zpvx krb5kdc[10270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533126, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:12:06.920661 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:06.906779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:43489 (local address 127.23.97.254:39137)
0812 11:12:06.906910 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:06.906914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:06.907713 (+ 799us) server_negotiation.cc:407] Connection header received
0812 11:12:06.909513 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:06.909516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:06.909565 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:06.909658 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:06.911146 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.911676 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.913118 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.913349 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.913468 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:06.916248 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:06.916255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:06.916258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:06.916288 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:06.918220 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.918959 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.918962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.918964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.919098 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:06.919549 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:06.919552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:06.919554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:06.919767 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:06.919829 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:06.920348 (+ 519us) server_negotiation.cc:299] Negotiation successful
0812 11:12:06.920488 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:12:06.921411 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:06.907133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39137 (local address 127.23.97.194:43489)
0812 11:12:06.907555 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:06.907595 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:06.909286 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:06.909811 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:06.909818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:06.910310 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:06.910967 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:06.910990 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.911825 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.911831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:06.912983 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:06.912994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:06.913503 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:06.913507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:06.913668 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:06.914235 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:06.914251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:06.916052 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:06.918352 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:06.918358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:06.918372 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:06.918838 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:06.919215 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:06.919218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:06.919220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:06.919421 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:06.919899 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:06.919904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:06.920105 (+ 201us) client_negotiation.cc:769] Sending connection context
0812 11:12:06.920299 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 11:12:06.920626 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20260812 11:12:06.922643 11005 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39137
I20260812 11:12:06.922963 11005 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:12:06.923530 11005 heartbeater.cc:511] Master 127.23.97.254:39137 requested a full tablet report, sending...
I20260812 11:12:06.924675 10359 ts_manager.cc:194] Registered new tserver with Master: 31c688ad0af442209215aee0f9fbc7a9 (127.23.97.194:33035)
I20260812 11:12:06.925348 10359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:43489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:12:06.940201 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:07.053691 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:07.053929 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:07.053970 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:07.058176 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:07.058231 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:07.058290 11014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:12:07.063398 11014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:07.064375 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:07.065515 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:07.075526 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:07.075551 11035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:07.075680 11014 server_base.cc:1034] running on GCE node
W20260812 11:12:07.075845 11028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:07.076342 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:07.076989 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:07.078240 11014 hybrid_clock.cc:648] HybridClock initialized: now 1786533127078206 us; error 54 us; skew 500 ppm
Aug 12 11:12:07 dist-test-slave-zpvx krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533127, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:07.081147 11014 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:12:07.081987 11014 webserver.cc:460] Webserver started at http://127.23.97.195:39093/ using document root <none> and password file <none>
I20260812 11:12:07.082516 11014 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:07.082576 11014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:07.082739 11014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:12:07.084803 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "eabcf6a65c064e9394070efbceb36d42"
format_stamp: "Formatted at 2026-08-12 11:12:07 on dist-test-slave-zpvx"
server_key: "c5a63a7013fbf63c664f1553fd3e2316"
server_key_iv: "78554c29d7dc0fc77d7f081e5d3d871e"
server_key_version: "encryptionkey@0"
I20260812 11:12:07.085331 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "eabcf6a65c064e9394070efbceb36d42"
format_stamp: "Formatted at 2026-08-12 11:12:07 on dist-test-slave-zpvx"
server_key: "c5a63a7013fbf63c664f1553fd3e2316"
server_key_iv: "78554c29d7dc0fc77d7f081e5d3d871e"
server_key_version: "encryptionkey@0"
I20260812 11:12:07.089112 11014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:12:07.091506 11045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:07.092803 11014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:12:07.092897 11014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "eabcf6a65c064e9394070efbceb36d42"
format_stamp: "Formatted at 2026-08-12 11:12:07 on dist-test-slave-zpvx"
server_key: "c5a63a7013fbf63c664f1553fd3e2316"
server_key_iv: "78554c29d7dc0fc77d7f081e5d3d871e"
server_key_version: "encryptionkey@0"
I20260812 11:12:07.092976 11014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:07.119357 11014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:07.122376 11014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:07.122530 11014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:07.123044 11014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:07.123800 11014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:07.123847 11014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:07.123886 11014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:07.123907 11014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:07.133104 11014 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:35351
I20260812 11:12:07.133137 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:35351 every 8 connection(s)
I20260812 11:12:07.133955 11014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:12:07.136451 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 11014
I20260812 11:12:07.136581 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:12:07.136922 23943 external_mini_cluster.cc:1428] Setting key ef8c105a39d1dc164c653f79d714093c
Aug 12 11:12:07 dist-test-slave-zpvx krb5kdc[10270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533127, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:07.148618 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.135816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54455 (local address 127.23.97.254:39137)
0812 11:12:07.135966 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.135970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.136964 (+ 994us) server_negotiation.cc:407] Connection header received
0812 11:12:07.138678 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.138681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.138737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.138818 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:07.140391 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.141056 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.142203 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.142430 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.142542 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.145047 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:07.145052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:07.145054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:07.145079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:07.146499 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:07.147075 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:07.147078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:07.147080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:07.147174 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:07.147521 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:07.147525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:07.147527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:07.147717 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:07.147830 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.148298 (+ 468us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.148446 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:12:07.149381 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.136186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39137 (local address 127.23.97.195:54455)
0812 11:12:07.136813 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.136854 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.138464 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.139012 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.139022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.139428 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:07.140195 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.140214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.141193 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.141196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.142072 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.142082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.142579 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.142593 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.142770 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.143376 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:07.143393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:07.144883 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:07.146668 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:07.146673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:07.146687 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:07.146956 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:07.147270 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:07.147274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:07.147276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:07.147418 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:07.147828 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:07.147835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:07.148063 (+ 228us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.148253 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.148594 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":224,"threads_started":1}
I20260812 11:12:07.150575 11281 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39137
I20260812 11:12:07.150823 11281 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:07.151360 11281 heartbeater.cc:511] Master 127.23.97.254:39137 requested a full tablet report, sending...
I20260812 11:12:07.152443 10359 ts_manager.cc:194] Registered new tserver with Master: eabcf6a65c064e9394070efbceb36d42 (127.23.97.195:35351)
I20260812 11:12:07.153021 10359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:54455
I20260812 11:12:07.162532 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:12:07.172253 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.164471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.23.97.254:39137)
0812 11:12:07.164586 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.164590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.164786 (+ 196us) server_negotiation.cc:407] Connection header received
0812 11:12:07.164895 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.164898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.164951 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.165018 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:07.165757 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.166355 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.167274 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.167512 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.167640 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.169046 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:07.169049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:07.169051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:07.169121 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:07.170485 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:07.170959 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:07.170964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:07.170966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:07.171062 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:07.171350 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:07.171354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:07.171356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:07.171579 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:07.171725 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.171961 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.172076 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:12:07.175216 10359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:12:07.178535 10359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:12:07.193245 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.187640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.23.97.193:39455)
0812 11:12:07.187921 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.187941 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.187968 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:12:07.188045 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.188050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.188536 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.189040 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:07.189215 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.190407 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.191917 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.192546 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.192669 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.192826 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.192992 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.193094 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":124,"threads_started":1}
I20260812 11:12:07.193233 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.187191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:39455 (local address 127.0.0.1:45684)
0812 11:12:07.187460 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.187475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.187557 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.188719 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.188725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.188747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:07.188878 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.188884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.190553 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.190557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.191763 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.191775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.192689 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.192693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.192798 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.192835 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.192952 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.193076 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":44,"thread_start_us":112,"threads_started":1}
I20260812 11:12:07.193403 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.187535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:33035 (local address 127.0.0.1:42158)
0812 11:12:07.187795 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.187812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.187906 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.189089 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.189092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.189112 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:07.189191 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.189200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.190584 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.190587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.191686 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.191695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.193013 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.193017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.193117 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.193135 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.193210 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.193285 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":49,"thread_start_us":136,"threads_started":1}
I20260812 11:12:07.193646 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.186784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35351 (local address 127.0.0.1:49242)
0812 11:12:07.187166 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.187218 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.187379 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.188862 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.188865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.188886 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:07.188984 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.188990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.190087 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.190092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.191780 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.191785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.192885 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.192889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.192980 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.193026 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.193077 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.193144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":75,"thread_start_us":94,"threads_started":1}
I20260812 11:12:07.193753 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.186935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.23.97.195:35351)
0812 11:12:07.188227 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.188241 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.188267 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:12:07.188346 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.188350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.188606 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.188896 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:07.189143 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.189952 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.191874 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.192702 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.193304 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.193444 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.193501 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.193601 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":163,"threads_started":1}
I20260812 11:12:07.193918 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.188036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.23.97.194:33035)
0812 11:12:07.188586 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.188602 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.188626 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:12:07.188704 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.188709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.188888 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.189038 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:07.189313 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.190409 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.191811 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.192658 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.192780 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.192947 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.193659 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.193752 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":116,"threads_started":1}
I20260812 11:12:07.196595 10626 tablet_service.cc:1467] Processing CreateTablet for tablet 8972e01aa6fa48829420d3513089ed96 (DEFAULT_TABLE table=test-table [id=df0b990f35ed489bb0817e9fd1de7694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:07.196805 11147 tablet_service.cc:1467] Processing CreateTablet for tablet 8972e01aa6fa48829420d3513089ed96 (DEFAULT_TABLE table=test-table [id=df0b990f35ed489bb0817e9fd1de7694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:07.196987 10871 tablet_service.cc:1467] Processing CreateTablet for tablet 8972e01aa6fa48829420d3513089ed96 (DEFAULT_TABLE table=test-table [id=df0b990f35ed489bb0817e9fd1de7694]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:07.197686 11147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8972e01aa6fa48829420d3513089ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:07.197736 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8972e01aa6fa48829420d3513089ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:07.197875 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8972e01aa6fa48829420d3513089ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:07.203752 11318 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Bootstrap starting.
I20260812 11:12:07.205446 11319 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Bootstrap starting.
I20260812 11:12:07.206089 11318 tablet_bootstrap.cc:654] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:07.207199 11318 log.cc:824] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:07.207592 11319 tablet_bootstrap.cc:654] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:07.208458 11319 log.cc:824] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:07.209033 11321 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: Bootstrap starting.
I20260812 11:12:07.209360 11318 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: No bootstrap required, opened a new log
I20260812 11:12:07.209553 11318 ts_tablet_manager.cc:1397] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:12:07.210255 11319 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: No bootstrap required, opened a new log
I20260812 11:12:07.210460 11319 ts_tablet_manager.cc:1397] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:12:07.211086 11321 tablet_bootstrap.cc:654] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:07.212329 11321 log.cc:824] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:07.213555 11318 raft_consensus.cc:348] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.213820 11318 raft_consensus.cc:374] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:07.213685 11319 raft_consensus.cc:348] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.213882 11318 raft_consensus.cc:729] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabcf6a65c064e9394070efbceb36d42, State: Initialized, Role: FOLLOWER
I20260812 11:12:07.213930 11319 raft_consensus.cc:374] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:07.213994 11319 raft_consensus.cc:729] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c688ad0af442209215aee0f9fbc7a9, State: Initialized, Role: FOLLOWER
I20260812 11:12:07.214275 11321 tablet_bootstrap.cc:492] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: No bootstrap required, opened a new log
I20260812 11:12:07.214365 11318 consensus_queue.cc:260] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.214457 11321 ts_tablet_manager.cc:1397] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:12:07.214457 11319 consensus_queue.cc:260] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.215359 11318 ts_tablet_manager.cc:1428] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260812 11:12:07.215435 11319 ts_tablet_manager.cc:1428] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:12:07.215791 11005 heartbeater.cc:503] Master 127.23.97.254:39137 was elected leader, sending a full tablet report...
I20260812 11:12:07.215803 11281 heartbeater.cc:503] Master 127.23.97.254:39137 was elected leader, sending a full tablet report...
I20260812 11:12:07.218098 11321 raft_consensus.cc:348] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.218360 11321 raft_consensus.cc:374] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:07.218411 11321 raft_consensus.cc:729] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba018a18b1c848e0a2a900889a98c464, State: Initialized, Role: FOLLOWER
I20260812 11:12:07.218816 11321 consensus_queue.cc:260] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.219724 11321 ts_tablet_manager.cc:1428] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:12:07.219780 10742 heartbeater.cc:503] Master 127.23.97.254:39137 was elected leader, sending a full tablet report...
I20260812 11:12:07.227587 11329 raft_consensus.cc:482] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:12:07.227759 11329 raft_consensus.cc:504] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.228750 11329 leader_election.cc:288] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eabcf6a65c064e9394070efbceb36d42 (127.23.97.195:35351), ba018a18b1c848e0a2a900889a98c464 (127.23.97.193:39455)
I20260812 11:12:07.232237 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.229215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:41043 (local address 127.23.97.195:35351)
0812 11:12:07.229337 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.229340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.229350 (+ 10us) server_negotiation.cc:407] Connection header received
0812 11:12:07.229387 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.229389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.229424 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.229517 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:07.229782 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.230274 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.231293 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.231693 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.231822 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.231871 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.232060 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.232126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:12:07.232239 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.228997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35351 (local address 127.23.97.194:41043)
0812 11:12:07.229108 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.229125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.229211 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.229530 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.229533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.229551 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:07.229665 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.229670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.230388 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.230393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.231159 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.231167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.231817 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.231821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.231925 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.231954 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.232044 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.232116 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 11:12:07.232702 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.228997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:39455 (local address 127.23.97.194:59477)
0812 11:12:07.229575 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.229586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:07.229700 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:07.229946 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:07.229949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:07.229965 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:07.230081 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.230086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.230801 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.230804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.231549 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:07.231557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.232173 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.232177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:07.232281 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.232330 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:12:07.232485 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:12:07.232552 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":304,"thread_start_us":103,"threads_started":1}
I20260812 11:12:07.232808 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.229104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:59477 (local address 127.23.97.193:39455)
0812 11:12:07.229236 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.229240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.229666 (+ 426us) server_negotiation.cc:407] Connection header received
0812 11:12:07.229806 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.229810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.229855 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.229935 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:07.230194 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.230680 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.231674 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.232052 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.232175 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.232219 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.232622 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.232694 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 11:12:07.232858 11183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8972e01aa6fa48829420d3513089ed96" candidate_uuid: "31c688ad0af442209215aee0f9fbc7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcf6a65c064e9394070efbceb36d42" is_pre_election: true
I20260812 11:12:07.233187 11183 raft_consensus.cc:2454] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c688ad0af442209215aee0f9fbc7a9 in term 0.
I20260812 11:12:07.233310 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8972e01aa6fa48829420d3513089ed96" candidate_uuid: "31c688ad0af442209215aee0f9fbc7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba018a18b1c848e0a2a900889a98c464" is_pre_election: true
I20260812 11:12:07.233558 10662 raft_consensus.cc:2454] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c688ad0af442209215aee0f9fbc7a9 in term 0.
I20260812 11:12:07.233664 10794 leader_election.cc:302] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c688ad0af442209215aee0f9fbc7a9, eabcf6a65c064e9394070efbceb36d42; no voters:
I20260812 11:12:07.233983 11329 raft_consensus.cc:2781] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:12:07.234093 11329 raft_consensus.cc:482] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:12:07.234150 11329 raft_consensus.cc:3037] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:07.235165 11329 raft_consensus.cc:504] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.235497 11329 leader_election.cc:288] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [CANDIDATE]: Term 1 election: Requested vote from peers eabcf6a65c064e9394070efbceb36d42 (127.23.97.195:35351), ba018a18b1c848e0a2a900889a98c464 (127.23.97.193:39455)
I20260812 11:12:07.235810 11183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8972e01aa6fa48829420d3513089ed96" candidate_uuid: "31c688ad0af442209215aee0f9fbc7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcf6a65c064e9394070efbceb36d42"
I20260812 11:12:07.235948 11183 raft_consensus.cc:3037] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:07.235958 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8972e01aa6fa48829420d3513089ed96" candidate_uuid: "31c688ad0af442209215aee0f9fbc7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba018a18b1c848e0a2a900889a98c464"
I20260812 11:12:07.236063 10662 raft_consensus.cc:3037] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:07.237268 10662 raft_consensus.cc:2454] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c688ad0af442209215aee0f9fbc7a9 in term 1.
I20260812 11:12:07.237268 11183 raft_consensus.cc:2454] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c688ad0af442209215aee0f9fbc7a9 in term 1.
I20260812 11:12:07.237542 10794 leader_election.cc:302] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c688ad0af442209215aee0f9fbc7a9, eabcf6a65c064e9394070efbceb36d42; no voters:
I20260812 11:12:07.237763 11329 raft_consensus.cc:2781] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:07.237931 11329 raft_consensus.cc:686] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 1 LEADER]: Becoming Leader. State: Replica: 31c688ad0af442209215aee0f9fbc7a9, State: Running, Role: LEADER
I20260812 11:12:07.238217 11329 consensus_queue.cc:237] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } }
I20260812 11:12:07.241091 10359 catalog_manager.cc:5495] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c688ad0af442209215aee0f9fbc7a9 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "31c688ad0af442209215aee0f9fbc7a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c688ad0af442209215aee0f9fbc7a9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 33035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20260812 11:12:07.254185 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.251036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.23.97.194:33035)
0812 11:12:07.251152 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:07.251155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:07.251168 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:12:07.251280 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:07.251283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:07.251331 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:07.251419 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:12:07.251632 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.252117 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.252983 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:07.253182 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:07.253273 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:07.253527 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:12:07.253885 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:12:07.253971 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:07.254025 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 11:12:07.254070 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:12:07.259588 10662 raft_consensus.cc:1261] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 1 FOLLOWER]: Refusing update from remote peer 31c688ad0af442209215aee0f9fbc7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:12:07.259591 11183 raft_consensus.cc:1261] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 1 FOLLOWER]: Refusing update from remote peer 31c688ad0af442209215aee0f9fbc7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:12:07.260293 11334 consensus_queue.cc:1035] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba018a18b1c848e0a2a900889a98c464" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:12:07.260522 11329 consensus_queue.cc:1035] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabcf6a65c064e9394070efbceb36d42" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:12:07.267736 11338 mvcc.cc:205] Tried to move back new op lower bound from 7317639689249021952 to 7317639689170272256. Current Snapshot: MvccSnapshot[applied={T|T < 7317639689249021952}]
I20260812 11:12:07.268013 11339 mvcc.cc:205] Tried to move back new op lower bound from 7317639689249021952 to 7317639689170272256. Current Snapshot: MvccSnapshot[applied={T|T < 7317639689249021952}]
I20260812 11:12:07.272490 11342 mvcc.cc:205] Tried to move back new op lower bound from 7317639689249021952 to 7317639689170272256. Current Snapshot: MvccSnapshot[applied={T|T < 7317639689249021952}]
I20260812 11:12:07.277411 10359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:12:07.277606 10359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:12:07.279945 10359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e540c569c8214ac1af1b26a6175b525d: Sending DeleteTablet for 3 replicas of tablet 8972e01aa6fa48829420d3513089ed96
I20260812 11:12:07.280696 11147 tablet_service.cc:1514] Processing DeleteTablet for tablet 8972e01aa6fa48829420d3513089ed96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:07 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:49242
I20260812 11:12:07.280825 10626 tablet_service.cc:1514] Processing DeleteTablet for tablet 8972e01aa6fa48829420d3513089ed96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:07 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:45684
I20260812 11:12:07.280911 10871 tablet_service.cc:1514] Processing DeleteTablet for tablet 8972e01aa6fa48829420d3513089ed96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:07 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:42158
I20260812 11:12:07.281270 11353 tablet_replica.cc:330] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464: stopping tablet replica
I20260812 11:12:07.281342 11354 tablet_replica.cc:330] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: stopping tablet replica
I20260812 11:12:07.281576 11353 raft_consensus.cc:2229] T 8972e01aa6fa48829420d3513089ed96 P ba018a18b1c848e0a2a900889a98c464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:12:07.281600 11354 raft_consensus.cc:2229] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:12:07.281899 11351 tablet_replica.cc:330] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: stopping tablet replica
I20260812 11:12:07.281926 11354 raft_consensus.cc:2258] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:12:07.281955 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 10499
I20260812 11:12:07.282639 11351 raft_consensus.cc:2229] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:12:07.283003 11351 raft_consensus.cc:2258] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:12:07.290431 11354 ts_tablet_manager.cc:1905] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:12:07.293678 11354 ts_tablet_manager.cc:1918] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:12:07.293776 11354 log.cc:1196] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/8972e01aa6fa48829420d3513089ed96
I20260812 11:12:07.294065 11354 ts_tablet_manager.cc:1939] T 8972e01aa6fa48829420d3513089ed96 P 31c688ad0af442209215aee0f9fbc7a9: Deleting consensus metadata
I20260812 11:12:07.295177 10335 catalog_manager.cc:4841] TS 31c688ad0af442209215aee0f9fbc7a9 (127.23.97.194:33035): tablet 8972e01aa6fa48829420d3513089ed96 (table test-table [id=df0b990f35ed489bb0817e9fd1de7694]) successfully deleted
I20260812 11:12:07.299152 11351 ts_tablet_manager.cc:1905] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:12:07.302130 11351 ts_tablet_manager.cc:1918] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:12:07.302222 11351 log.cc:1196] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/8972e01aa6fa48829420d3513089ed96
I20260812 11:12:07.302510 11351 ts_tablet_manager.cc:1939] T 8972e01aa6fa48829420d3513089ed96 P eabcf6a65c064e9394070efbceb36d42: Deleting consensus metadata
I20260812 11:12:07.310375 10335 catalog_manager.cc:4841] TS eabcf6a65c064e9394070efbceb36d42 (127.23.97.195:35351): tablet 8972e01aa6fa48829420d3513089ed96 (table test-table [id=df0b990f35ed489bb0817e9fd1de7694]) successfully deleted
W20260812 11:12:07.318984 10332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:39455 (error 108)
I20260812 11:12:07.319114 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 10750
I20260812 11:12:07.320314 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:07.319908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:39455 (local address 127.0.0.1:45698)
0812 11:12:07.320149 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:07.320217 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260812 11:12:07.320497 10332 catalog_manager.cc:4584] TS ba018a18b1c848e0a2a900889a98c464 (127.23.97.193:39455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8972e01aa6fa48829420d3513089ed96: Network error: Client connection negotiation failed: client connection to 127.23.97.193:39455: connect: Connection refused (error 111)
I20260812 11:12:07.328872 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 11014
I20260812 11:12:07.347975 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 10299
2026-08-12T11:12:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2551 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:12:07 dist-test-slave-zpvx krb5kdc[11359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:12:07 dist-test-slave-zpvx krb5kdc[11359](info): set up 2 sockets
Aug 12 11:12:07 dist-test-slave-zpvx krb5kdc[11359](info): commencing operation
krb5kdc: starting...
W20260812 11:12:08.660763 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:12:08 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:12:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangeradmin_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangeradmin_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangerlookup_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangerlookup_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
I20260812 11:12:08.839424 23943 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/postgres -l logfile start
2026-08-12 11:12:11.347 UTC [11413] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:12:11.347 UTC [11413] LOG: listening on IPv4 address "127.23.97.212", port 47467
2026-08-12 11:12:11.351 UTC [11413] LOG: listening on Unix socket "/tmp/.s.PGSQL.47467"
2026-08-12 11:12:11.373 UTC [11421] LOG: database system was shut down at 2026-08-12 11:12:09 UTC
2026-08-12 11:12:11.377 UTC [11413] LOG: database system is ready to accept connections
I20260812 11:12:12.603408 23943 mini_postgres.cc:96] Postgres bound to 47467
2026-08-12 11:12:12.611 UTC [11434] FATAL: database "slave" does not exist
127.23.97.212:47467 - accepting connections
I20260812 11:12:12.612799 23943 mini_ranger.cc:161] Starting Ranger...
I20260812 11:12:12.629287 23943 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:12:13.108152 23943 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:12:13.108261 23943 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin
2026-08-12 11:12:13,456 [I] DB FLAVOR :POSTGRES
2026-08-12 11:12:13,456 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:12:13,457 [I] Checking connection..
2026-08-12 11:12:13,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:12:13,692 [I] Checking connection passed.
2026-08-12 11:12:13,692 [I] --------- Verifying version history table ---------
2026-08-12 11:12:13,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:12:13,964 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:12:13,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:12:14,213 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:12:14,214 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:12:14,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:12:14.474 UTC [11569] WARNING: there is no transaction in progress
2026-08-12 11:12:14,483 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:12:14,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:12:14,734 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:12:14,734 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:12:14,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:12:14,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:12:15,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-12 11:12:15,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:12:15,726 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:12:15,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:12:15,996 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:12:15,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:12:16,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:12:16,514 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:12:16,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:12:17.493 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.509 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.525 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.540 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.555 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.571 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.628 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.635 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.648 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.657 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.667 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.680 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.689 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.695 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.701 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:17.707 UTC [11795] WARNING: there is no transaction in progress
2026-08-12 11:12:18,322 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:12:18,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:12:18,580 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:12:18,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:12:18,822 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:12:18,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:12:19,088 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:12:19,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:12:19,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:12:19,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-12 11:12:19,841 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:12:19,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:12:20,095 [I] DB_PATCHES have already been applied
I20260812 11:12:20.101302 23943 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvWFrLZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:12:20.101387 23943 mini_ranger.cc:193] Using host: 127.23.97.212
I20260812 11:12:20.105930 23943 mini_ranger.cc:227] Ranger admin URL: http://127.23.97.212:53169
Aug 12, 2026 11:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:12:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:12:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:12:21.548454 23943 mini_ranger.cc:160] Time spent starting Ranger: real 8.936s user 0.004s sys 0.008s
Aug 12, 2026 11:12:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:12:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 11:12:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:12:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:12:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:12:43.213395 23943 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangerkms_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/rangerkms_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.97.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.97.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:12:43 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:12:43.301702 23943 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:12:43.312168 23943 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:12:44.155531 23943 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:12:44.155616 23943 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms
I20260812 11:12:44.155714 23943 mini_ranger_kms.cc:227] Using postgres at 127.23.97.212:47467
2026-08-12 11:12:44,466 [I] DB FLAVOR :POSTGRES
2026-08-12 11:12:44,466 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:12:44,466 [I] Checking connection
2026-08-12 11:12:44,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:12:44,777 [I] connection success
2026-08-12 11:12:44,777 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:12:44,777 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:12:44,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:12:45,106 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:12:45,107 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:12:45,107 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:12:45,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:47467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:12:45,382 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:12:45.849936 23943 mini_ranger_kms.cc:310] Created kms service
I20260812 11:12:46.034849 23943 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:12:46.080853 23943 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:12:46 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:46.498355 23943 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:12:46.498518 23943 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:12:46.498539 23943 mini_ranger_kms.cc:240] Using host: 127.23.97.212
I20260812 11:12:46.503317 23943 mini_ranger_kms.cc:276] Ranger KMS PID: 12139
I20260812 11:12:46.503463 23943 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.97.212:34873
Aug 12, 2026 11:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:12:49 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:49.422408 23943 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.121s user 0.006s sys 0.280s
I20260812 11:12:49.422551 23943 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:12:49.422672 23943 mini_ranger_kms.cc:401] 127.23.97.212:34873/kms/v1/keys
Aug 12 11:12:49 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
Aug 12 11:12:50 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
Aug 12 11:12:52 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:12:52.651772 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39171
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:39171
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:34873/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:52.784497 12232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:52.784772 12232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:52.784816 12232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:52.789177 12232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:12:52.789245 12232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:52.789265 12232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:52.789281 12232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:12:52.789297 12232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:12:52.795507 12232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.97.212:34873/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:39171
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:52.796795 12232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:52.797910 12232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:52.809406 12244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:52.809377 12238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:52.809410 12241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:52.809383 12232 server_base.cc:1034] running on GCE node
I20260812 11:12:52.810309 12232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:52.811420 12232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:52.812579 12232 hybrid_clock.cc:648] HybridClock initialized: now 1786533172812547 us; error 45 us; skew 500 ppm
Aug 12 11:12:52 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:52.818898 12232 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:12:52.819921 12232 webserver.cc:460] Webserver started at http://127.23.97.254:41389/ using document root <none> and password file <none>
I20260812 11:12:52.820575 12232 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:52.820634 12232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:52.820858 12232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:12:52 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:52.867641 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "0dc2f97569e24dfd83d814aa5890e6dd"
format_stamp: "Formatted at 2026-08-12 11:12:52 on dist-test-slave-zpvx"
server_key: "d0ade7e67249457476da6645d7a80b03"
server_key_iv: "ba9722b54a967852fea3974b0f27fc71"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:52.868394 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "0dc2f97569e24dfd83d814aa5890e6dd"
format_stamp: "Formatted at 2026-08-12 11:12:52 on dist-test-slave-zpvx"
server_key: "d0ade7e67249457476da6645d7a80b03"
server_key_iv: "ba9722b54a967852fea3974b0f27fc71"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:52.872539 12232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:12:52.888687 12258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:52.890427 12232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:12:52.890592 12232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "0dc2f97569e24dfd83d814aa5890e6dd"
format_stamp: "Formatted at 2026-08-12 11:12:52 on dist-test-slave-zpvx"
server_key: "d0ade7e67249457476da6645d7a80b03"
server_key_iv: "ba9722b54a967852fea3974b0f27fc71"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:52.890746 12232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:52.941583 12232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:52.944650 12232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:52.944866 12232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:52.957472 12232 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:39171
I20260812 11:12:52.957605 12358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:39171 every 8 connection(s)
I20260812 11:12:52.958523 12232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:12:52.959969 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 12232
I20260812 11:12:52.960106 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:12:52.960472 23943 external_mini_cluster.cc:1428] Setting key fa87cdcc58636f5e5cf04c6ffd822129
I20260812 11:12:52.962181 12359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:52.969995 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: Bootstrap starting.
I20260812 11:12:52.973301 12359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: Neither blocks nor log segments found. Creating new log.
Aug 12 11:12:52 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:52.974599 12359 log.cc:824] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:52.977221 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: No bootstrap required, opened a new log
I20260812 11:12:52.980682 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:52.962125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.23.97.254:39171)
0812 11:12:52.962756 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:52.962813 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:52.962861 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:12:52.964219 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:52.964246 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:52.964720 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:52.965102 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:52.966252 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:52.968289 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:52.969666 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:52.970489 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:52.970659 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:52.974120 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:52.974128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:52.974145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:52.974178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:52.977115 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:52.977703 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:52.977710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:52.977716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:52.977857 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:52.978206 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:52.978209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:52.978211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:52.978731 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:52.978920 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:52.979335 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 11:12:52.979636 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":191,"threads_started":1}
I20260812 11:12:52.981587 12359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } }
I20260812 11:12:52.981827 12359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:52.981902 12359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc2f97569e24dfd83d814aa5890e6dd, State: Initialized, Role: FOLLOWER
I20260812 11:12:52.982383 12359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } }
I20260812 11:12:52.982550 12359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:12:52.982623 12359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:12:52.982749 12359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:52.983788 12359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } }
I20260812 11:12:52.984123 12359 leader_election.cc:302] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dc2f97569e24dfd83d814aa5890e6dd; no voters:
I20260812 11:12:52.984505 12359 leader_election.cc:288] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:12:52.984578 12367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:52.985186 12367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [term 1 LEADER]: Becoming Leader. State: Replica: 0dc2f97569e24dfd83d814aa5890e6dd, State: Running, Role: LEADER
I20260812 11:12:52.985574 12367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } }
I20260812 11:12:52.985929 12359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:12:52.987506 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } } }
I20260812 11:12:52.987650 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:52.988312 12380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:12:52.990485 12371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dc2f97569e24dfd83d814aa5890e6dd. Latest consensus state: current_term: 1 leader_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc2f97569e24dfd83d814aa5890e6dd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39171 } } }
I20260812 11:12:52.990702 12371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd [sys.catalog]: This master's current role is: LEADER
I20260812 11:12:52.993450 12380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:12:52.994048 12359 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:12:53.007910 12380 catalog_manager.cc:1347] Generated new cluster ID: f5bb89b82d164e4cade3b895b064a1ae
I20260812 11:12:53.007997 12380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:12:53.041208 12380 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:12:53.042693 12380 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:12:53.066537 12380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: Generated new TSK 0
I20260812 11:12:53.067323 12380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:12:53.951292 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:34873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:54.102166 12452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:54.102494 12452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:54.102582 12452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:54.107177 12452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:54.107273 12452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:54.107374 12452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:12:54.114491 12452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.97.212:34873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:54.115855 12452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:54.117237 12452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:54.128736 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:54.128769 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:54.129213 12452 server_base.cc:1034] running on GCE node
W20260812 11:12:54.129338 12474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:54.129757 12452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:54.130404 12452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:54.131606 12452 hybrid_clock.cc:648] HybridClock initialized: now 1786533174131571 us; error 53 us; skew 500 ppm
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:54.135025 12452 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:12:54.136144 12452 webserver.cc:460] Webserver started at http://127.23.97.193:44239/ using document root <none> and password file <none>
I20260812 11:12:54.136920 12452 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:54.137023 12452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:54.137286 12452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:54.152752 12452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9bac813ba7ba481ea3f701e0b8f27c63"
server_key_iv: "0756eb8c60ff882f5c2939f164a4f8c1"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.153488 12452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9bac813ba7ba481ea3f701e0b8f27c63"
server_key_iv: "0756eb8c60ff882f5c2939f164a4f8c1"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.158013 12452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 11:12:54.173138 12486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.174573 12452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:12:54.174716 12452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9bac813ba7ba481ea3f701e0b8f27c63"
server_key_iv: "0756eb8c60ff882f5c2939f164a4f8c1"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.174854 12452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:54.199476 12452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:54.205224 12452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:54.205940 12452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:54.206988 12452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:54.208227 12452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:54.208351 12452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.208868 12452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:54.209103 12452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.230774 12452 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:35371
I20260812 11:12:54.232517 12452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:12:54.232775 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:35371 every 8 connection(s)
I20260812 11:12:54.239571 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 12452
I20260812 11:12:54.239717 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:12:54.240046 23943 external_mini_cluster.cc:1428] Setting key b186ab118d90623489dd2bca92d85649
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:54.259886 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:54.235537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:57217 (local address 127.23.97.254:39171)
0812 11:12:54.235782 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:54.235787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:54.240590 (+ 4803us) server_negotiation.cc:407] Connection header received
0812 11:12:54.240878 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:54.240882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:54.240943 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:54.241031 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:54.247742 (+ 6711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.248659 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.250456 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.250827 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.250929 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:54.254919 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:54.254923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:54.254926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:54.254955 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:54.257017 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:54.257735 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:54.257739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:54.257741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:54.257842 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:54.258422 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:54.258426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:54.258428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:54.258697 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:54.258793 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:54.259524 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 11:12:54.259691 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:12:54.262297 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:54.236025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39171 (local address 127.23.97.193:57217)
0812 11:12:54.237622 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:54.237694 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:54.239164 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:54.245642 (+ 6478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:54.246306 (+ 664us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:54.246762 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:54.247512 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:54.247536 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.248814 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.248817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:54.250282 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:54.250314 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.251153 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.251158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:54.251412 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:54.252327 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:54.252348 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:54.254710 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:54.257173 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:54.257179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:54.257196 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:54.257553 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:54.257973 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:54.257979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:54.257981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:54.258271 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:54.258873 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:54.258879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:54.259215 (+ 336us) client_negotiation.cc:769] Sending connection context
0812 11:12:54.260437 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0812 11:12:54.260849 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:12:54.264082 12699 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39171
I20260812 11:12:54.264345 12699 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:54.265091 12699 heartbeater.cc:511] Master 127.23.97.254:39171 requested a full tablet report, sending...
I20260812 11:12:54.267437 12290 ts_manager.cc:194] Registered new tserver with Master: eca8fc3f4ef045ea9bf36c66d8fdc66a (127.23.97.193:35371)
I20260812 11:12:54.269151 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:57217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:12:54.300024 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:34873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 11:12:54.517956 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:54.518311 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:54.518376 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:54.524456 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:54.524531 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:54.524626 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:12:54.531778 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.97.212:34873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:54.533030 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:54.534166 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:12:54.547173 12730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:54.547300 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:54.547391 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:54.547600 12706 server_base.cc:1034] running on GCE node
I20260812 11:12:54.548105 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:54.548864 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:54.550050 12706 hybrid_clock.cc:648] HybridClock initialized: now 1786533174550037 us; error 45 us; skew 500 ppm
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:54.554301 12706 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:12:54.555352 12706 webserver.cc:460] Webserver started at http://127.23.97.194:43939/ using document root <none> and password file <none>
I20260812 11:12:54.556028 12706 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:54.556116 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:54.556351 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:54.582628 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "91b3dd12ea8249768b1a5c8a36916840"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9ca15155057496cc8af2f9dfd108307f"
server_key_iv: "3872c8ff1cd81f5d9cc7e6134891fafd"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.583453 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "91b3dd12ea8249768b1a5c8a36916840"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9ca15155057496cc8af2f9dfd108307f"
server_key_iv: "3872c8ff1cd81f5d9cc7e6134891fafd"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.588469 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 11:12:54.607951 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.612900 12706 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260812 11:12:54.613178 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "91b3dd12ea8249768b1a5c8a36916840"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "9ca15155057496cc8af2f9dfd108307f"
server_key_iv: "3872c8ff1cd81f5d9cc7e6134891fafd"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.613440 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:12:54.656440 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:54.660809 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:54.661124 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:54.662235 12706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:54.663647 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:54.663765 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.663899 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:54.663980 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:54.684984 12706 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:34617
I20260812 11:12:54.686372 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:12:54.689589 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:34617 every 8 connection(s)
I20260812 11:12:54.695133 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 12706
I20260812 11:12:54.695273 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:12:54.695590 23943 external_mini_cluster.cc:1428] Setting key b68b7b7f2f5ebce6a0d8d3f5fb221a55
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:54.710492 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:54.694817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:36059 (local address 127.23.97.254:39171)
0812 11:12:54.694955 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:54.694961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:54.694979 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:12:54.696144 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:54.696148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:54.696209 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:54.696285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:54.698183 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.698997 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.700996 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.701437 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.701529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:54.705456 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:54.705474 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:54.705478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:54.705518 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:54.707480 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:54.708201 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:54.708206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:54.708208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:54.708341 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:54.708835 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:54.708838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:54.708840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:54.709172 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:54.709266 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:54.710125 (+ 859us) server_negotiation.cc:299] Negotiation successful
0812 11:12:54.710302 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:12:54.712499 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:54.693633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39171 (local address 127.23.97.194:36059)
0812 11:12:54.694542 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:54.694605 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:54.695882 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:54.696777 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:54.696787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:54.697189 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:54.697965 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:54.698004 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.699157 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.699161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:54.700814 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:54.700827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:54.701740 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:54.701744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:54.701959 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:54.702747 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:54.702771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:54.705182 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:54.707666 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:54.707672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:54.707686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:54.708046 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:54.708497 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:54.708500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:54.708502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:54.708711 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:54.709367 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:54.709374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:54.709751 (+ 377us) client_negotiation.cc:769] Sending connection context
0812 11:12:54.710724 (+ 973us) client_negotiation.cc:240] Negotiation successful
0812 11:12:54.711149 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":202,"threads_started":1}
I20260812 11:12:54.714196 12961 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39171
I20260812 11:12:54.714458 12961 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:54.715041 12961 heartbeater.cc:511] Master 127.23.97.254:39171 requested a full tablet report, sending...
I20260812 11:12:54.716490 12290 ts_manager.cc:194] Registered new tserver with Master: 91b3dd12ea8249768b1a5c8a36916840 (127.23.97.194:34617)
I20260812 11:12:54.717507 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:36059
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:12:54.744179 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:34873/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:12:54.937794 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:12:54.938139 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:12:54.938205 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:12:54.944288 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:12:54.944429 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:12:54.944551 12969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:12:54.952538 12969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.97.212:34873/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:12:54.954052 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:12:54.955443 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:12:54.970468 12969 server_base.cc:1034] running on GCE node
W20260812 11:12:54.970759 12976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:54.971150 12977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:12:54.971715 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:12:54.972227 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:12:54.973033 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:12:54.974243 12969 hybrid_clock.cc:648] HybridClock initialized: now 1786533174974219 us; error 38 us; skew 500 ppm
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:12:54.978997 12969 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:12:54.980255 12969 webserver.cc:460] Webserver started at http://127.23.97.195:39553/ using document root <none> and password file <none>
I20260812 11:12:54.981088 12969 fs_manager.cc:359] Metadata directory not provided
I20260812 11:12:54.981248 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:12:54.981570 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:12:54 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:54.998346 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "7b34397a37de4ff1bad8741e388d8b1e"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "e414b0be7881ab3e1b1fb845e73bf919"
server_key_iv: "abc1b6aae8f65e805f7e34dec2fe5161"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:54.999194 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "7b34397a37de4ff1bad8741e388d8b1e"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "e414b0be7881ab3e1b1fb845e73bf919"
server_key_iv: "abc1b6aae8f65e805f7e34dec2fe5161"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:55.019652 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.004s sys 0.000s
I20260812 11:12:55.039565 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:55.041424 12969 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:12:55.041646 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "7b34397a37de4ff1bad8741e388d8b1e"
format_stamp: "Formatted at 2026-08-12 11:12:54 on dist-test-slave-zpvx"
server_key: "e414b0be7881ab3e1b1fb845e73bf919"
server_key_iv: "abc1b6aae8f65e805f7e34dec2fe5161"
server_key_version: "kuduclusterkey@0"
I20260812 11:12:55.041878 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 11:12:55 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:12:55.096814 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:12:55.101466 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:12:55.101727 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:12:55.102936 12969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:12:55.104771 12969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:12:55.104899 12969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:55.105037 12969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:12:55.105109 12969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:12:55.121426 12969 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:35431
I20260812 11:12:55.123066 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:12:55.125590 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:35431 every 8 connection(s)
I20260812 11:12:55.129822 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 12969
I20260812 11:12:55.129958 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:12:55.130306 23943 external_mini_cluster.cc:1428] Setting key ce3e9a9452ab81143135926fcd11d333
Aug 12 11:12:55 dist-test-slave-zpvx krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533174, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:12:55.140491 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.128954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:48595 (local address 127.23.97.254:39171)
0812 11:12:55.129104 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.129109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.129127 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:12:55.129830 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.129837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.129905 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.129969 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:55.131888 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.132739 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.133912 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.134135 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.134253 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.136794 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:55.136797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:55.136799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:55.136823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:55.138392 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:55.138936 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:55.138941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:55.138943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:55.139026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:55.139383 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:55.139388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:55.139389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:55.139571 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:55.139690 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.140172 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.140298 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:12:55.141417 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.127715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:39171 (local address 127.23.97.195:48595)
0812 11:12:55.128311 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.128405 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.129544 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.130405 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.130415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.130943 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:12:55.131725 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.131749 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.132874 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.132876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.133773 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.133783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.134292 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.134296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.134479 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.135122 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:12:55.135138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:12:55.136623 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:12:55.138534 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:55.138540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:55.138554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:55.138830 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:55.139123 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:12:55.139127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:12:55.139129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:12:55.139285 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:12:55.139679 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:12:55.139686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:12:55.139936 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.140128 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.140471 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":159,"threads_started":1}
I20260812 11:12:55.142721 13225 heartbeater.cc:347] Connected to a master server at 127.23.97.254:39171
I20260812 11:12:55.142961 13225 heartbeater.cc:464] Registering TS with master...
I20260812 11:12:55.143532 13225 heartbeater.cc:511] Master 127.23.97.254:39171 requested a full tablet report, sending...
I20260812 11:12:55.144697 12290 ts_manager.cc:194] Registered new tserver with Master: 7b34397a37de4ff1bad8741e388d8b1e (127.23.97.195:35431)
I20260812 11:12:55.145256 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:48595
I20260812 11:12:55.155804 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:12:55.174702 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.158355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.23.97.254:39171)
0812 11:12:55.165476 (+ 7121us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.165481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.165497 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:12:55.165541 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.165544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.165592 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.165688 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:12:55.166731 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.167500 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.168754 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.169080 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.169205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.171057 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:12:55.171061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:12:55.171064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:12:55.171092 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:12:55.172875 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:55.173367 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:55.173371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:55.173373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:55.173466 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:12:55.173765 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:12:55.173769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:12:55.173771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:12:55.173984 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:12:55.174122 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.174389 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.174531 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7027}
I20260812 11:12:55.178088 12290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:12:55.182557 12290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:12:55.207924 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.194833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:34617 (local address 127.0.0.1:60910)
0812 11:12:55.196342 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.196410 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.196574 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.200998 (+ 4424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.201004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.201047 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:55.201238 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.201255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.203504 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.203506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.205809 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.205819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.207401 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.207404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.207510 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.207547 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.207652 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.207771 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":156,"threads_started":1}
I20260812 11:12:55.208992 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.195007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.23.97.194:34617)
0812 11:12:55.200354 (+ 5347us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.200411 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.200435 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:12:55.200511 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.200515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.200705 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.200863 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:55.201407 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.203301 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.206044 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.207061 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.207168 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.207336 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.208707 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.208823 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5241,"thread_start_us":131,"threads_started":1}
I20260812 11:12:55.210391 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.196156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35431 (local address 127.0.0.1:53972)
0812 11:12:55.197435 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.197447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.197516 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.203437 (+ 5921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.203440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.203459 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:55.204605 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.204615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.206894 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.206897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.208565 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.208576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.209960 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.209965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.210092 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.210117 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.210189 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.210260 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":931,"thread_start_us":125,"threads_started":1}
I20260812 11:12:55.210580 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.196351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.23.97.195:35431)
0812 11:12:55.199847 (+ 3496us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.199870 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.199897 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:12:55.200227 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.200234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.200520 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.200717 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:55.205492 (+ 4775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.206710 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.208767 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.209651 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.209765 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.209895 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.210252 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.210347 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375,"thread_start_us":192,"threads_started":1}
I20260812 11:12:55.212646 12845 tablet_service.cc:1467] Processing CreateTablet for tablet ccc05ed6f3be4ba18351222f611a9455 (DEFAULT_TABLE table=test-table [id=56549c0391eb4e83b5b6d99e527d51c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:55.213997 12845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc05ed6f3be4ba18351222f611a9455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:55.213724 13097 tablet_service.cc:1467] Processing CreateTablet for tablet ccc05ed6f3be4ba18351222f611a9455 (DEFAULT_TABLE table=test-table [id=56549c0391eb4e83b5b6d99e527d51c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:55.214778 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc05ed6f3be4ba18351222f611a9455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:55.222007 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.194836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:35371 (local address 127.0.0.1:41690)
0812 11:12:55.197134 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.197148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.197260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.201555 (+ 4295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.201558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.201575 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:55.201707 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.201713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.203252 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.203255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.205357 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.205368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.221550 (+ 16182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.221554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.221695 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.221724 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.221782 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.221862 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2242,"mutex_wait_us":695,"thread_start_us":132,"threads_started":1}
I20260812 11:12:55.222798 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.195040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.23.97.193:35371)
0812 11:12:55.201130 (+ 6090us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.201136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.201156 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:12:55.201236 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.201241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.201408 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.201857 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:55.201989 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.203104 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.214743 (+ 11639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.215569 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.222320 (+ 6751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.222488 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.222540 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.222640 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5986,"thread_start_us":82,"threads_started":1}
I20260812 11:12:55.226187 12579 tablet_service.cc:1467] Processing CreateTablet for tablet ccc05ed6f3be4ba18351222f611a9455 (DEFAULT_TABLE table=test-table [id=56549c0391eb4e83b5b6d99e527d51c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:12:55.227427 12579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc05ed6f3be4ba18351222f611a9455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:12:55.228912 13261 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: Bootstrap starting.
I20260812 11:12:55.231865 13261 tablet_bootstrap.cc:654] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:55.233682 13261 log.cc:824] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:55.234627 13265 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Bootstrap starting.
I20260812 11:12:55.238032 13265 tablet_bootstrap.cc:654] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:55.239388 13265 log.cc:824] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:55.243288 13265 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: No bootstrap required, opened a new log
I20260812 11:12:55.243372 13261 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: No bootstrap required, opened a new log
I20260812 11:12:55.243496 13265 ts_tablet_manager.cc:1397] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260812 11:12:55.243590 13261 ts_tablet_manager.cc:1397] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260812 11:12:55.245224 13270 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: Bootstrap starting.
I20260812 11:12:55.248922 13270 tablet_bootstrap.cc:654] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: Neither blocks nor log segments found. Creating new log.
I20260812 11:12:55.249701 13265 raft_consensus.cc:348] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.249977 13261 raft_consensus.cc:348] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.250239 13270 log.cc:824] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:12:55.250223 13265 raft_consensus.cc:374] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:55.250351 13265 raft_consensus.cc:729] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b34397a37de4ff1bad8741e388d8b1e, State: Initialized, Role: FOLLOWER
I20260812 11:12:55.250268 13261 raft_consensus.cc:374] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:55.250466 13261 raft_consensus.cc:729] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b3dd12ea8249768b1a5c8a36916840, State: Initialized, Role: FOLLOWER
I20260812 11:12:55.251015 13261 consensus_queue.cc:260] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.251107 13265 consensus_queue.cc:260] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.252458 13261 ts_tablet_manager.cc:1428] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260812 11:12:55.252678 13270 tablet_bootstrap.cc:492] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: No bootstrap required, opened a new log
I20260812 11:12:55.252887 13270 ts_tablet_manager.cc:1397] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:12:55.253598 13265 ts_tablet_manager.cc:1428] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260812 11:12:55.253829 12961 heartbeater.cc:503] Master 127.23.97.254:39171 was elected leader, sending a full tablet report...
I20260812 11:12:55.254046 13225 heartbeater.cc:503] Master 127.23.97.254:39171 was elected leader, sending a full tablet report...
I20260812 11:12:55.258775 13270 raft_consensus.cc:348] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.259122 13270 raft_consensus.cc:374] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:12:55.259239 13270 raft_consensus.cc:729] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca8fc3f4ef045ea9bf36c66d8fdc66a, State: Initialized, Role: FOLLOWER
I20260812 11:12:55.259783 13270 consensus_queue.cc:260] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.261025 13270 ts_tablet_manager.cc:1428] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:12:55.261427 12699 heartbeater.cc:503] Master 127.23.97.254:39171 was elected leader, sending a full tablet report...
I20260812 11:12:55.297267 13274 raft_consensus.cc:482] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:12:55.297547 13274 raft_consensus.cc:504] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.298993 13274 leader_election.cc:288] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eca8fc3f4ef045ea9bf36c66d8fdc66a (127.23.97.193:35371), 91b3dd12ea8249768b1a5c8a36916840 (127.23.97.194:34617)
I20260812 11:12:55.304909 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.299334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:35371 (local address 127.23.97.195:54795)
0812 11:12:55.299662 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.299679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.299783 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.300706 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.300709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.300743 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:55.300899 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.300908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.302005 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.302009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.303334 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.303347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.304450 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.304454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.304556 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.304580 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.304668 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.304740 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":7}
I20260812 11:12:55.305804 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.299920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:58667 (local address 127.23.97.194:34617)
0812 11:12:55.300037 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.300042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.300456 (+ 414us) server_negotiation.cc:407] Connection header received
0812 11:12:55.300674 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.300677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.300728 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.300779 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:55.301418 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.302208 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.304155 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.304851 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.304946 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.304991 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.305611 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.305686 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:12:55.305807 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.300017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54795 (local address 127.23.97.193:35371)
0812 11:12:55.300188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.300193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.300209 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:12:55.300264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.300266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.300337 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.300451 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:12:55.301055 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.301865 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.303564 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.304175 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.304306 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.304361 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.305556 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.305665 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:12:55.306282 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.299432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:34617 (local address 127.23.97.195:58667)
0812 11:12:55.300304 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.300319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.300537 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.300921 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:12:55.300924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:12:55.301020 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:12:55.301271 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.301280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.302359 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.302362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.303997 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:12:55.304010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.305278 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.305283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:12:55.305440 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.305464 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:12:55.305987 (+ 523us) client_negotiation.cc:240] Negotiation successful
0812 11:12:55.306063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":3,"thread_start_us":198,"threads_started":1}
I20260812 11:12:55.306507 12610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc05ed6f3be4ba18351222f611a9455" candidate_uuid: "7b34397a37de4ff1bad8741e388d8b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" is_pre_election: true
I20260812 11:12:55.306876 12610 raft_consensus.cc:2454] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b34397a37de4ff1bad8741e388d8b1e in term 0.
I20260812 11:12:55.306924 12885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc05ed6f3be4ba18351222f611a9455" candidate_uuid: "7b34397a37de4ff1bad8741e388d8b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b3dd12ea8249768b1a5c8a36916840" is_pre_election: true
I20260812 11:12:55.307211 12885 raft_consensus.cc:2454] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b34397a37de4ff1bad8741e388d8b1e in term 0.
I20260812 11:12:55.307822 13003 leader_election.cc:302] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b34397a37de4ff1bad8741e388d8b1e, 91b3dd12ea8249768b1a5c8a36916840; no voters:
I20260812 11:12:55.308317 13274 raft_consensus.cc:2781] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:12:55.308450 13274 raft_consensus.cc:482] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:12:55.308511 13274 raft_consensus.cc:3037] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:55.310773 13274 raft_consensus.cc:504] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.311208 13274 leader_election.cc:288] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [CANDIDATE]: Term 1 election: Requested vote from peers eca8fc3f4ef045ea9bf36c66d8fdc66a (127.23.97.193:35371), 91b3dd12ea8249768b1a5c8a36916840 (127.23.97.194:34617)
I20260812 11:12:55.311759 12885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc05ed6f3be4ba18351222f611a9455" candidate_uuid: "7b34397a37de4ff1bad8741e388d8b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b3dd12ea8249768b1a5c8a36916840"
I20260812 11:12:55.311787 12610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc05ed6f3be4ba18351222f611a9455" candidate_uuid: "7b34397a37de4ff1bad8741e388d8b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a"
I20260812 11:12:55.311895 12885 raft_consensus.cc:3037] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:55.311937 12610 raft_consensus.cc:3037] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:12:55.313907 12885 raft_consensus.cc:2454] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b34397a37de4ff1bad8741e388d8b1e in term 1.
I20260812 11:12:55.314070 12610 raft_consensus.cc:2454] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b34397a37de4ff1bad8741e388d8b1e in term 1.
I20260812 11:12:55.314385 13003 leader_election.cc:302] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b34397a37de4ff1bad8741e388d8b1e, 91b3dd12ea8249768b1a5c8a36916840; no voters:
I20260812 11:12:55.314685 13274 raft_consensus.cc:2781] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:12:55.314918 13274 raft_consensus.cc:686] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 1 LEADER]: Becoming Leader. State: Replica: 7b34397a37de4ff1bad8741e388d8b1e, State: Running, Role: LEADER
I20260812 11:12:55.315292 13274 consensus_queue.cc:237] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } }
I20260812 11:12:55.319183 12290 catalog_manager.cc:5495] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b34397a37de4ff1bad8741e388d8b1e (127.23.97.195). New cstate: current_term: 1 leader_uuid: "7b34397a37de4ff1bad8741e388d8b1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b34397a37de4ff1bad8741e388d8b1e" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35431 } health_report { overall_health: HEALTHY } } }
I20260812 11:12:55.368165 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.361685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.23.97.195:35431)
0812 11:12:55.361826 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 11:12:55.361832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:12:55.361848 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:12:55.361948 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:12:55.361951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:12:55.362026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:12:55.362165 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:12:55.362664 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.363546 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.364951 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:12:55.365401 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:12:55.365539 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:12:55.366851 (+ 1312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:12:55.367606 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:12:55.367779 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:12:55.367878 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 11:12:55.367946 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:12:55.376044 12610 raft_consensus.cc:1261] T ccc05ed6f3be4ba18351222f611a9455 P eca8fc3f4ef045ea9bf36c66d8fdc66a [term 1 FOLLOWER]: Refusing update from remote peer 7b34397a37de4ff1bad8741e388d8b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:12:55.376052 12885 raft_consensus.cc:1261] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 1 FOLLOWER]: Refusing update from remote peer 7b34397a37de4ff1bad8741e388d8b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:12:55.376914 13277 consensus_queue.cc:1035] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca8fc3f4ef045ea9bf36c66d8fdc66a" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:12:55.377343 13277 consensus_queue.cc:1035] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b3dd12ea8249768b1a5c8a36916840" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:12:55.377357 13227 tablet.cc:2369] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:12:55.393368 13302 mvcc.cc:205] Tried to move back new op lower bound from 7317639886331559936 to 7317639886095163392. Current Snapshot: MvccSnapshot[applied={T|T < 7317639886331559936}]
I20260812 11:12:55.400020 13283 mvcc.cc:205] Tried to move back new op lower bound from 7317639886331559936 to 7317639886095163392. Current Snapshot: MvccSnapshot[applied={T|T < 7317639886331559936}]
I20260812 11:12:55.410195 12290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:12:55.410390 12290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:12:55.412917 12290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dc2f97569e24dfd83d814aa5890e6dd: Sending DeleteTablet for 3 replicas of tablet ccc05ed6f3be4ba18351222f611a9455
I20260812 11:12:55.414048 12579 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc05ed6f3be4ba18351222f611a9455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:55 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41690
I20260812 11:12:55.414224 12845 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc05ed6f3be4ba18351222f611a9455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:55 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:60910
I20260812 11:12:55.414338 13097 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc05ed6f3be4ba18351222f611a9455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:12:55 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:53972
I20260812 11:12:55.414865 13325 tablet_replica.cc:330] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: stopping tablet replica
I20260812 11:12:55.414908 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 12452
I20260812 11:12:55.421114 13325 raft_consensus.cc:2229] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:12:55.421728 13325 raft_consensus.cc:2258] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:12:55.424536 13325 ts_tablet_manager.cc:1905] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:12:55.427798 13325 ts_tablet_manager.cc:1918] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:12:55.428028 13325 log.cc:1196] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/ccc05ed6f3be4ba18351222f611a9455
I20260812 11:12:55.428573 13325 ts_tablet_manager.cc:1939] T ccc05ed6f3be4ba18351222f611a9455 P 7b34397a37de4ff1bad8741e388d8b1e: Deleting consensus metadata
I20260812 11:12:55.430004 12262 catalog_manager.cc:4841] TS 7b34397a37de4ff1bad8741e388d8b1e (127.23.97.195:35431): tablet ccc05ed6f3be4ba18351222f611a9455 (table test-table [id=56549c0391eb4e83b5b6d99e527d51c8]) successfully deleted
I20260812 11:12:55.436216 13324 tablet_replica.cc:330] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840: stopping tablet replica
W20260812 11:12:55.436511 12263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:35371 (error 108)
I20260812 11:12:55.436796 13324 raft_consensus.cc:2229] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:12:55.437247 13324 raft_consensus.cc:2258] T ccc05ed6f3be4ba18351222f611a9455 P 91b3dd12ea8249768b1a5c8a36916840 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 11:12:55.438772 13258 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:12:55.438099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:35371 (local address 127.0.0.1:40850)
0812 11:12:55.438235 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.438253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:12:55.438354 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:12:55.438649 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:35371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 11:12:55.438941 12263 catalog_manager.cc:4584] TS eca8fc3f4ef045ea9bf36c66d8fdc66a (127.23.97.193:35371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccc05ed6f3be4ba18351222f611a9455: Network error: Client connection negotiation failed: client connection to 127.23.97.193:35371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:12:55.440750 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 12706
I20260812 11:12:55.473760 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.473343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:34617 (local address 127.0.0.1:58510)
0812 11:12:55.473546 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.473649 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.194:34617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260812 11:12:55.474597 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 12969
I20260812 11:12:55.492882 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.492440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:35371 (local address 127.0.0.1:40862)
0812 11:12:55.492681 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.492762 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 11:12:55.493813 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:12:55.493516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:35371 (local address 127.0.0.1:40874)
0812 11:12:55.493645 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 11:12:55.493715 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 11:12:55.496861 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 12232
I20260812 11:12:55.514930 23943 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:12:55.860312 23943 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:12:55.860524 23943 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:12:56.205911 23943 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:12:56.206 UTC [11413] LOG: received smart shutdown request
2026-08-12 11:12:56.210 UTC [11413] LOG: background worker "logical replication launcher" (PID 11431) exited with exit code 1
2026-08-12 11:12:56.210 UTC [11424] LOG: shutting down
2026-08-12 11:12:56.371 UTC [11413] LOG: database system is shut down
2026-08-12T11:12:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49152 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:12:56 dist-test-slave-zpvx krb5kdc[13344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:12:56 dist-test-slave-zpvx krb5kdc[13344](info): set up 2 sockets
Aug 12 11:12:56 dist-test-slave-zpvx krb5kdc[13344](info): commencing operation
krb5kdc: starting...
W20260812 11:12:57.820995 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:12:57 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:12:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangeradmin_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangeradmin_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangerlookup_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangerlookup_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
I20260812 11:12:57.993363 23943 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/postgres -l logfile start
2026-08-12 11:13:00.493 UTC [13405] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:13:00.493 UTC [13405] LOG: listening on IPv4 address "127.23.97.212", port 33353
2026-08-12 11:13:00.497 UTC [13405] LOG: listening on Unix socket "/tmp/.s.PGSQL.33353"
2026-08-12 11:13:00.518 UTC [13411] LOG: database system was shut down at 2026-08-12 11:12:59 UTC
2026-08-12 11:13:00.522 UTC [13405] LOG: database system is ready to accept connections
I20260812 11:13:01.843649 23943 mini_postgres.cc:96] Postgres bound to 33353
2026-08-12 11:13:01.851 UTC [13424] FATAL: database "slave" does not exist
127.23.97.212:33353 - accepting connections
I20260812 11:13:01.852058 23943 mini_ranger.cc:161] Starting Ranger...
I20260812 11:13:01.867622 23943 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:13:02.352535 23943 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:13:02.352612 23943 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin
2026-08-12 11:13:02,613 [I] DB FLAVOR :POSTGRES
2026-08-12 11:13:02,613 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:13:02,613 [I] Checking connection..
2026-08-12 11:13:02,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:13:02,852 [I] Checking connection passed.
2026-08-12 11:13:02,853 [I] --------- Verifying version history table ---------
2026-08-12 11:13:02,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:13:03,084 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:13:03,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:13:03,355 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:13:03,356 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:13:03,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:13:03.583 UTC [13554] WARNING: there is no transaction in progress
2026-08-12 11:13:03,592 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:13:03,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:13:03,863 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:13:03,863 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:13:03,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:13:04,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:13:04,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-12 11:13:04,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:13:04,900 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:13:04,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:13:05,159 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:13:05,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:13:05,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:13:05,693 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:13:05,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:13:06.651 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.667 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.682 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.699 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.714 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.734 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.787 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.794 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.808 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.818 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.828 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.842 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.852 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.858 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.865 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:06.872 UTC [13752] WARNING: there is no transaction in progress
2026-08-12 11:13:07,503 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:13:07,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:13:07,762 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:13:07,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:13:08,033 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:13:08,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:13:08,300 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:13:08,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:13:08,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:13:08,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-12 11:13:09,065 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:13:09,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:13:09,316 [I] DB_PATCHES have already been applied
I20260812 11:13:09.322196 23943 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvWFrLZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:13:09.322263 23943 mini_ranger.cc:193] Using host: 127.23.97.212
I20260812 11:13:09.326661 23943 mini_ranger.cc:227] Ranger admin URL: http://127.23.97.212:39741
Aug 12, 2026 11:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:13:10.856705 23943 mini_ranger.cc:160] Time spent starting Ranger: real 9.005s user 0.000s sys 0.012s
Aug 12, 2026 11:13:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:13:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 11:13:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:13:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:13:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:13:31.483042 23943 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.97.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.97.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangerkms_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.97.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/rangerkms_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.97.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.97.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.97.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/HTTP_127.23.97.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:13:31 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533211, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:13:31.580909 23943 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:13:31.591761 23943 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:13:32.445505 23943 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:13:32.445601 23943 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms
I20260812 11:13:32.445621 23943 mini_ranger_kms.cc:227] Using postgres at 127.23.97.212:33353
2026-08-12 11:13:32,620 [I] DB FLAVOR :POSTGRES
2026-08-12 11:13:32,620 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:13:32,620 [I] Checking connection
2026-08-12 11:13:32,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:13:32,867 [I] connection success
2026-08-12 11:13:32,867 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:13:32,868 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:13:32,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:13:33,103 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:13:33,103 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:13:33,104 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:13:33,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.97.212:33353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:13:33,382 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:13:33.836814 23943 mini_ranger_kms.cc:310] Created kms service
I20260812 11:13:34.017809 23943 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:13:34.067166 23943 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:13:34 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533211, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:34.499220 23943 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:13:34.499449 23943 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvWFrLZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvWFrLZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:13:34.499476 23943 mini_ranger_kms.cc:240] Using host: 127.23.97.212
I20260812 11:13:34.504204 23943 mini_ranger_kms.cc:276] Ranger KMS PID: 14116
I20260812 11:13:34.504288 23943 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.97.212:38149
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:13:36 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 11:13:37 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:37.449908 23943 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.869s user 0.000s sys 0.160s
I20260812 11:13:37.450031 23943 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:13:37.450091 23943 mini_ranger_kms.cc:401] 127.23.97.212:38149/kms/v1/keys
Aug 12 11:13:37 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.97.212@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
Aug 12 11:13:39 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:13:40.007995 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43517
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43517
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:40.121932 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:40.122181 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:40.122221 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:40.126307 14208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:40.126358 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:40.126377 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:40.126394 14208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:40.126436 14208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:40.131677 14208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.97.212:38149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43517
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:40.132725 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:40.133713 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:40.142561 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:40.142633 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:40.142633 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:40.142699 14208 server_base.cc:1034] running on GCE node
I20260812 11:13:40.143492 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:40.144690 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:40.145884 14208 hybrid_clock.cc:648] HybridClock initialized: now 1786533220145856 us; error 39 us; skew 500 ppm
Aug 12 11:13:40 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533220, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:40.148878 14208 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:40.149776 14208 webserver.cc:460] Webserver started at http://127.23.97.254:36221/ using document root <none> and password file <none>
I20260812 11:13:40.150323 14208 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:40.150408 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:40.150643 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:13:40 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533220, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:40.206277 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "c1e99afa9bb944e3901826080dd194bd"
format_stamp: "Formatted at 2026-08-12 11:13:40 on dist-test-slave-zpvx"
server_key: "078920d52515c25a7ff8d377b27a8864"
server_key_iv: "b935aa1f70b47f29da15c4afc8139eae"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:40.207015 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "c1e99afa9bb944e3901826080dd194bd"
format_stamp: "Formatted at 2026-08-12 11:13:40 on dist-test-slave-zpvx"
server_key: "078920d52515c25a7ff8d377b27a8864"
server_key_iv: "b935aa1f70b47f29da15c4afc8139eae"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:40.211313 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:13:40.229801 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:40.231236 14208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:13:40.231424 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c1e99afa9bb944e3901826080dd194bd"
format_stamp: "Formatted at 2026-08-12 11:13:40 on dist-test-slave-zpvx"
server_key: "078920d52515c25a7ff8d377b27a8864"
server_key_iv: "b935aa1f70b47f29da15c4afc8139eae"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:40.231566 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:40.253155 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:40.255978 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:40.256592 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:40.263945 14208 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43517
I20260812 11:13:40.263952 14350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43517 every 8 connection(s)
I20260812 11:13:40.265027 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:40.266542 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 14208
I20260812 11:13:40.266644 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:13:40.266961 23943 external_mini_cluster.cc:1428] Setting key 2da30aff0f3fe87055d2f95d9850a24e
I20260812 11:13:40.268309 14354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:40.275506 14354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Bootstrap starting.
Aug 12 11:13:40 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:40.278103 14354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:40.279145 14354 log.cc:824] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:40.281632 14354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: No bootstrap required, opened a new log
I20260812 11:13:40.283792 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:40.268504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.23.97.254:43517)
0812 11:13:40.268933 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:40.268952 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:40.268996 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:13:40.269884 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:40.269907 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:40.270331 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:40.270699 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:40.271414 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:40.272923 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:40.274206 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:40.274895 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:40.275049 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:40.277936 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:40.277946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:40.277965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:40.278004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:40.280864 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:40.281349 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:40.281356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:40.281362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:40.281494 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:40.281842 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:40.281846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:40.281848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:40.282270 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:40.282482 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:40.282802 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 11:13:40.283069 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260812 11:13:40.284806 14354 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:40.285046 14354 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:40.285133 14354 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e99afa9bb944e3901826080dd194bd, State: Initialized, Role: FOLLOWER
I20260812 11:13:40.285583 14354 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:40.285740 14354 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:40.285821 14354 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:40.285918 14354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:40.286852 14354 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:40.287192 14354 leader_election.cc:302] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e99afa9bb944e3901826080dd194bd; no voters:
I20260812 11:13:40.287569 14354 leader_election.cc:288] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:13:40.287707 14365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:40.287983 14365 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 LEADER]: Becoming Leader. State: Replica: c1e99afa9bb944e3901826080dd194bd, State: Running, Role: LEADER
I20260812 11:13:40.288352 14365 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:40.288890 14354 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:40.290155 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e99afa9bb944e3901826080dd194bd. Latest consensus state: current_term: 1 leader_uuid: "c1e99afa9bb944e3901826080dd194bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } } }
I20260812 11:13:40.290295 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:40.290385 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e99afa9bb944e3901826080dd194bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } } }
I20260812 11:13:40.290498 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:40.290726 14383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:40.293845 14383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:40.294241 14354 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:13:40.305029 14383 catalog_manager.cc:1347] Generated new cluster ID: 73de5b2f3fca46499bb55d24c4d93943
I20260812 11:13:40.305110 14383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:40.323134 14383 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:13:40.324306 14383 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:40.334441 14383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Generated new TSK 0
I20260812 11:13:40.335318 14383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:13:41.031526 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:41.182498 14451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:41.182739 14451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:41.182782 14451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:41.186901 14451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:41.186982 14451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:41.187083 14451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:13:41.192899 14451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:41.193953 14451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:41.194981 14451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:41.207089 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:41.207757 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:41.208025 14451 server_base.cc:1034] running on GCE node
W20260812 11:13:41.207090 14473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:41.208564 14451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:41.209405 14451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:41.210609 14451 hybrid_clock.cc:648] HybridClock initialized: now 1786533221210575 us; error 43 us; skew 500 ppm
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:41.213848 14451 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:13:41.214978 14451 webserver.cc:460] Webserver started at http://127.23.97.193:35461/ using document root <none> and password file <none>
I20260812 11:13:41.215721 14451 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:41.215819 14451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:41.216092 14451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:41.234781 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "bc7974f191fe4748a7d22d2ee403ef55"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "a89e3ef6bd309acc2842a09b73dcd409"
server_key_iv: "0b2cd0d435b53e8287dc6c083af04669"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.237685 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "bc7974f191fe4748a7d22d2ee403ef55"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "a89e3ef6bd309acc2842a09b73dcd409"
server_key_iv: "0b2cd0d435b53e8287dc6c083af04669"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.243170 14451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:13:41.260353 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.262207 14451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:13:41.262354 14451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "bc7974f191fe4748a7d22d2ee403ef55"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "a89e3ef6bd309acc2842a09b73dcd409"
server_key_iv: "0b2cd0d435b53e8287dc6c083af04669"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.262504 14451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:41.280195 14451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:41.283757 14451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:41.283959 14451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:41.284672 14451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:41.285733 14451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:41.285818 14451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.285893 14451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:41.285936 14451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.296654 14451 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:45729
I20260812 11:13:41.297839 14451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:13:41.305500 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 14451
I20260812 11:13:41.305635 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:13:41.305945 23943 external_mini_cluster.cc:1428] Setting key 82b414dc971ab0e602688ab159f6fe23
I20260812 11:13:41.306677 14722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:45729 every 8 connection(s)
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:13:41.323518 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:41.307601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:48869 (local address 127.23.97.254:43517)
0812 11:13:41.309252 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:41.309262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:41.310544 (+ 1282us) server_negotiation.cc:407] Connection header received
0812 11:13:41.310988 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:41.310992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:41.311047 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:41.311145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:41.312738 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.313615 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.315317 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.315622 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.315753 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:41.319040 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:41.319044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:41.319047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:41.319075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:41.321053 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:41.321675 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:41.321678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:41.321679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:41.321758 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:41.322112 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:41.322120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:41.322122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:41.322303 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:41.322415 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:41.322977 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 11:13:41.323104 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":97,"threads_started":1}
I20260812 11:13:41.324314 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:41.308078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.193:48869)
0812 11:13:41.309539 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:41.309599 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:41.310733 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:41.311464 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:41.311473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:41.311841 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:41.312557 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:41.312578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.313768 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.313772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:41.315153 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:41.315166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.315804 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.315809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:41.316042 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:41.316825 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:41.316861 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:41.318827 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:41.321185 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:41.321191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:41.321215 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:41.321549 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:41.321855 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:41.321859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:41.321861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:41.322002 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:41.322413 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:41.322420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:41.322723 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 11:13:41.322919 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 11:13:41.323255 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":167,"threads_started":1}
I20260812 11:13:41.327134 14726 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:41.327399 14726 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:41.328006 14726 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:41.330062 14270 ts_manager.cc:194] Registered new tserver with Master: bc7974f191fe4748a7d22d2ee403ef55 (127.23.97.193:45729)
I20260812 11:13:41.331797 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:48869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:13:41.343422 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 11:13:41.470427 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:41.470731 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:41.470837 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:41.475289 14742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:41.475382 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:41.475488 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:13:41.481702 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:41.483006 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:41.484414 14742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:41.494988 14762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:41.495052 14751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:41.495404 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:41.495996 14742 server_base.cc:1034] running on GCE node
I20260812 11:13:41.496447 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:41.497084 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:41.504462 14742 hybrid_clock.cc:648] HybridClock initialized: now 1786533221504399 us; error 95 us; skew 500 ppm
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:41.507802 14742 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:13:41.508956 14742 webserver.cc:460] Webserver started at http://127.23.97.194:45109/ using document root <none> and password file <none>
I20260812 11:13:41.509770 14742 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:41.510217 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:41.510522 14742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:41.529197 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "fc29a9e899cb47e880067d1ff4136fa7"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "e29a5ae23b4bd214a48a114df08d9824"
server_key_iv: "b28330a2ae26ef51911c4a43b01059fe"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.530041 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "fc29a9e899cb47e880067d1ff4136fa7"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "e29a5ae23b4bd214a48a114df08d9824"
server_key_iv: "b28330a2ae26ef51911c4a43b01059fe"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.538102 14742 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s
I20260812 11:13:41.560580 14774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.562234 14742 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:13:41.562390 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "fc29a9e899cb47e880067d1ff4136fa7"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "e29a5ae23b4bd214a48a114df08d9824"
server_key_iv: "b28330a2ae26ef51911c4a43b01059fe"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.562543 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:41.606470 14742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:41.610752 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:41.610992 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:41.611940 14742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:41.613230 14742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:41.613298 14742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.613354 14742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:41.613384 14742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.636405 14742 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:43823
I20260812 11:13:41.636433 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:43823 every 8 connection(s)
I20260812 11:13:41.638098 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:13:41.643204 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 14742
I20260812 11:13:41.643373 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:13:41.643827 23943 external_mini_cluster.cc:1428] Setting key c8b070c81161f83e8ea03b67daa7b20e
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:41.662739 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:41.641098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:59431 (local address 127.23.97.254:43517)
0812 11:13:41.641240 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:41.641245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:41.642709 (+ 1464us) server_negotiation.cc:407] Connection header received
0812 11:13:41.644632 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:41.644635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:41.644695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:41.644750 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:41.646834 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.647704 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.649520 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.649999 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.650138 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:41.657533 (+ 7395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:41.657538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:41.657541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:41.657588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:41.659479 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:41.660248 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:41.660252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:41.660271 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:41.660427 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:41.661003 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:41.661007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:41.661008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:41.661336 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:41.661446 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:41.662396 (+ 950us) server_negotiation.cc:299] Negotiation successful
0812 11:13:41.662548 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:13:41.664700 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:41.641661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.194:59431)
0812 11:13:41.642517 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:41.642597 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:41.644106 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:41.645163 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:41.645171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:41.645919 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:41.646652 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:41.646674 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.647869 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.647872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:41.649359 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:41.649371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:41.650194 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:41.650198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:41.650443 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:41.655294 (+ 4851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:41.655318 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:41.657292 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:41.659682 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:41.659689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:41.659708 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:41.660085 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:41.660586 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:41.660590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:41.660592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:41.660867 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:41.661523 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:41.661530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:41.661965 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 11:13:41.663199 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0812 11:13:41.663578 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":228,"threads_started":1}
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:13:41.666419 15004 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:41.666703 15004 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:41.667301 15004 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:41.669318 14270 ts_manager.cc:194] Registered new tserver with Master: fc29a9e899cb47e880067d1ff4136fa7 (127.23.97.194:43823)
I20260812 11:13:41.670090 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:59431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:13:41.695487 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:41.865200 15010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:41.865490 15010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:41.865578 15010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:41.872349 15010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:41.872493 15010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:41.872604 15010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:13:41.879365 15010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:41.880748 15010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:41.882067 15010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:41.894804 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:41.895156 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:41.895274 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:41.895538 15010 server_base.cc:1034] running on GCE node
I20260812 11:13:41.895942 15010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:41.896746 15010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:41.897928 15010 hybrid_clock.cc:648] HybridClock initialized: now 1786533221897894 us; error 47 us; skew 500 ppm
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:41.901422 15010 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:13:41.902455 15010 webserver.cc:460] Webserver started at http://127.23.97.195:43615/ using document root <none> and password file <none>
I20260812 11:13:41.903044 15010 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:41.903142 15010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:41.903370 15010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:41.925710 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "d9273b27d5e54a21b9ee34768c4ae242"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "507f250bd65b3f07a8da81556674f6ea"
server_key_iv: "2ad0428ff18a6b9d56e2f09a48030b30"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.926808 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "d9273b27d5e54a21b9ee34768c4ae242"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "507f250bd65b3f07a8da81556674f6ea"
server_key_iv: "2ad0428ff18a6b9d56e2f09a48030b30"
server_key_version: "kuduclusterkey@0"
Aug 12 11:13:41 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:41.953521 15010 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.004s sys 0.000s
I20260812 11:13:41.985124 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:41.987032 15010 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260812 11:13:41.987205 15010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d9273b27d5e54a21b9ee34768c4ae242"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "507f250bd65b3f07a8da81556674f6ea"
server_key_iv: "2ad0428ff18a6b9d56e2f09a48030b30"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:41.987404 15010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:42.022796 15010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:42.026929 15010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:42.027225 15010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:42.028045 15010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:42.029337 15010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:42.029451 15010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.029565 15010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:42.029640 15010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.050726 15010 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:37179
I20260812 11:13:42.053189 15010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:13:42.055658 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:37179 every 8 connection(s)
I20260812 11:13:42.063130 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 15010
I20260812 11:13:42.063265 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:13:42.063573 23943 external_mini_cluster.cc:1428] Setting key 7a550f21fc71152d82f0ab7f4c5edcc0
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533221, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:42.078696 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.059277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:53331 (local address 127.23.97.254:43517)
0812 11:13:42.059402 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:42.059407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:42.060404 (+ 997us) server_negotiation.cc:407] Connection header received
0812 11:13:42.061723 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:42.061727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:42.061783 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:42.061838 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:42.063710 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.064564 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.068512 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.068870 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.068951 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.072533 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:42.072537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:42.072540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:42.072611 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:42.074420 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.075064 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.075068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.075070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.075323 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.075797 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.075800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.075802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.076021 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:42.076125 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:42.078381 (+ 2256us) server_negotiation.cc:299] Negotiation successful
0812 11:13:42.078530 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:13:42.080641 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.058118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.195:53331)
0812 11:13:42.060219 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:42.060277 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:42.061469 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:42.062257 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:42.062265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:42.062781 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:42.063532 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.063552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.066581 (+ 3029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.066585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.068315 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.068327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.069176 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.069180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.069394 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.070205 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:42.070225 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:42.072321 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:42.074563 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.074569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.074584 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.074936 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.075479 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.075482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.075484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.075675 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.076198 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:42.076204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:42.076531 (+ 327us) client_negotiation.cc:769] Sending connection context
0812 11:13:42.079155 (+ 2624us) client_negotiation.cc:240] Negotiation successful
0812 11:13:42.079495 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"thread_start_us":172,"threads_started":1}
I20260812 11:13:42.082144 15267 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:42.082388 15267 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:42.083087 15267 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:42.084244 14270 ts_manager.cc:194] Registered new tserver with Master: d9273b27d5e54a21b9ee34768c4ae242 (127.23.97.195:37179)
I20260812 11:13:42.084985 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:53331
I20260812 11:13:42.089963 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:13:42.090788 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 14451
I20260812 11:13:42.156562 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 14742
I20260812 11:13:42.170112 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 15010
I20260812 11:13:42.184424 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 14208
I20260812 11:13:42.209431 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43517
--webserver_interface=127.23.97.254
--webserver_port=36221
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43517
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:42.322813 15270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:42.323061 15270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:42.323099 15270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:42.327208 15270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:42.327265 15270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:42.327284 15270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:42.327301 15270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:42.327315 15270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:42.332544 15270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43517
--ranger_config_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=36221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:42.333496 15270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:42.334439 15270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:42.343072 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.343073 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.343080 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:42.343107 15270 server_base.cc:1034] running on GCE node
I20260812 11:13:42.343843 15270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:42.344945 15270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:42.346158 15270 hybrid_clock.cc:648] HybridClock initialized: now 1786533222346114 us; error 44 us; skew 500 ppm
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:42.349094 15270 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:42.349989 15270 webserver.cc:460] Webserver started at http://127.23.97.254:36221/ using document root <none> and password file <none>
I20260812 11:13:42.350492 15270 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:42.350559 15270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:42.397621 15270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260812 11:13:42.400017 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.401317 15270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 11:13:42.401471 15270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c1e99afa9bb944e3901826080dd194bd"
format_stamp: "Formatted at 2026-08-12 11:13:40 on dist-test-slave-zpvx"
server_key: "078920d52515c25a7ff8d377b27a8864"
server_key_iv: "b935aa1f70b47f29da15c4afc8139eae"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:42.402043 15270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:42.414611 15270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:42.417490 15270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:42.417717 15270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:42.425372 15270 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43517
I20260812 11:13:42.425384 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43517 every 8 connection(s)
I20260812 11:13:42.426469 15270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:42.426916 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 15270
I20260812 11:13:42.427423 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:45729
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:43517
--webserver_port=35461
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:42.431047 15414 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:13:42.432803 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Bootstrap starting.
I20260812 11:13:42.449491 15414 log.cc:824] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:42.456036 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:13:42.456503 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Bootstrap complete.
I20260812 11:13:42.460613 15414 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:42.461030 15414 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e99afa9bb944e3901826080dd194bd, State: Initialized, Role: FOLLOWER
I20260812 11:13:42.461529 15414 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:42.461681 15414 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:42.461769 15414 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:42.461891 15414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:13:42.463757 15414 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:42.464089 15414 leader_election.cc:302] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e99afa9bb944e3901826080dd194bd; no voters:
I20260812 11:13:42.464507 15414 leader_election.cc:288] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:13:42.464591 15421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:13:42.464865 15421 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [term 2 LEADER]: Becoming Leader. State: Replica: c1e99afa9bb944e3901826080dd194bd, State: Running, Role: LEADER
I20260812 11:13:42.465224 15421 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } }
I20260812 11:13:42.465788 15414 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:42.466197 15421 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e99afa9bb944e3901826080dd194bd. Latest consensus state: current_term: 2 leader_uuid: "c1e99afa9bb944e3901826080dd194bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } } }
I20260812 11:13:42.466802 15421 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:42.467028 15425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1e99afa9bb944e3901826080dd194bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e99afa9bb944e3901826080dd194bd" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43517 } } }
I20260812 11:13:42.467155 15425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:42.468775 15438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:42.469149 15414 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:13:42.475878 15438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:42.476804 15438 catalog_manager.cc:1259] Loaded cluster ID: 73de5b2f3fca46499bb55d24c4d93943
I20260812 11:13:42.476907 15438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:42.479568 15438 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:42.480664 15438 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Loaded TSK: 0
I20260812 11:13:42.482064 15438 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:13:42.584000 15418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:42.584450 15418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:42.584573 15418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:42.590821 15418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:42.590972 15418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:42.591081 15418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:13:42.599102 15418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:45729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=35461
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:42.600418 15418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:42.601804 15418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:42.614711 15494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.615592 15489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.615976 15490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:42.616320 15418 server_base.cc:1034] running on GCE node
I20260812 11:13:42.616945 15418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:42.617754 15418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:42.618911 15418 hybrid_clock.cc:648] HybridClock initialized: now 1786533222618863 us; error 63 us; skew 500 ppm
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:42.622622 15418 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:13:42.623652 15418 webserver.cc:460] Webserver started at http://127.23.97.193:35461/ using document root <none> and password file <none>
I20260812 11:13:42.624490 15418 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:42.624619 15418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:42.645406 15418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:13:42.648451 15511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.649917 15418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:13:42.650076 15418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "bc7974f191fe4748a7d22d2ee403ef55"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "a89e3ef6bd309acc2842a09b73dcd409"
server_key_iv: "0b2cd0d435b53e8287dc6c083af04669"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:42.650635 15418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:42.669018 15418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:42.672513 15418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:42.672722 15418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:42.673445 15418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:42.674489 15418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:42.674557 15418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.674623 15418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:42.674655 15418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.686121 15418 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:45729
I20260812 11:13:42.687645 15418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:13:42.691732 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:45729 every 8 connection(s)
I20260812 11:13:42.695595 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 15418
I20260812 11:13:42.696116 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:43823
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:43517
--webserver_port=45109
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:42.717278 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.695239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.193:49771)
0812 11:13:42.696335 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:42.696404 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:42.697200 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:42.698813 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:42.698822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:42.699213 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:42.699881 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.699900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.701846 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.701853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.703441 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.703457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.704964 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.704972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.705184 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.705915 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:42.705941 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:42.707692 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:42.713706 (+ 6014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.713713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.713728 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.714067 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.714466 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.714469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.714471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.714647 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.715409 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:42.715415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:42.715718 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 11:13:42.715915 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 11:13:42.716236 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":114,"threads_started":1}
I20260812 11:13:42.717470 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.695127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:49771 (local address 127.23.97.254:43517)
0812 11:13:42.697589 (+ 2462us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:42.697606 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:42.697641 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:13:42.697864 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:42.697874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:42.698330 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:42.699675 (+ 1345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:42.700040 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.701665 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.703636 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.704502 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.704706 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.707953 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:42.707962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:42.707982 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:42.708023 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:42.713548 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.714209 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.714216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.714222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.714361 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.714754 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.714757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.714759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.715227 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:42.715344 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:42.716593 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0812 11:13:42.716837 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2321,"thread_start_us":138,"threads_started":1}
I20260812 11:13:42.720580 15749 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:42.720830 15749 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:42.721336 15749 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
W20260812 11:13:42.723501 15749 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:43517 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:13:42.723641 15749 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:42.723825 15749 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:42.724644 15749 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:42.724845 15749 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
W20260812 11:13:42.725592 15749 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:13:42.850199 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:42.850585 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:42.850667 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:42.855235 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:42.855320 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:42.855413 15757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:13:42.862526 15757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:43823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=45109
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:42.863646 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:42.864830 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:42.875676 15771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.875759 15770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:42.876082 15773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:42.876305 15757 server_base.cc:1034] running on GCE node
I20260812 11:13:42.876816 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:42.877645 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:42.881207 15757 hybrid_clock.cc:648] HybridClock initialized: now 1786533222881132 us; error 93 us; skew 500 ppm
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:42.884925 15757 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:13:42.886076 15757 webserver.cc:460] Webserver started at http://127.23.97.194:45109/ using document root <none> and password file <none>
I20260812 11:13:42.886826 15757 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:42.886915 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:42.921047 15757 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:13:42.923975 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.925715 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:13:42.925868 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "fc29a9e899cb47e880067d1ff4136fa7"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "e29a5ae23b4bd214a48a114df08d9824"
server_key_iv: "b28330a2ae26ef51911c4a43b01059fe"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:42.926599 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:42.949828 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:42.953656 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:42.953891 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:42.954650 15757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:42.955801 15757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:42.955869 15757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.955927 15757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:42.955960 15757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:42.967900 15757 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:43823
I20260812 11:13:42.970458 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:43823 every 8 connection(s)
I20260812 11:13:42.971719 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:13:42.978886 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 15757
I20260812 11:13:42.979416 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.97.212:38149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:37179
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:43517
--webserver_port=43615
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:13:42 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533222, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:42.996074 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.977520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:45661 (local address 127.23.97.254:43517)
0812 11:13:42.977667 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:42.977672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:42.978976 (+ 1304us) server_negotiation.cc:407] Connection header received
0812 11:13:42.980823 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:42.980831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:42.980901 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:42.980964 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:42.982526 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.983390 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.985282 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.985721 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.985832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.990990 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:42.990997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:42.991001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:42.991037 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:42.993092 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.993841 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.993847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.993852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.993957 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:42.994446 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:42.994453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:42.994457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:42.994741 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:42.994846 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:42.995660 (+ 814us) server_negotiation.cc:299] Negotiation successful
0812 11:13:42.995824 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:13:42.997860 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:42.978125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.194:45661)
0812 11:13:42.978780 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:42.978847 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:42.980544 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:42.981436 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:42.981449 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:42.981759 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:42.982366 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.982381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.983564 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.983570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.985122 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:42.985138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:42.986265 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:42.986269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:42.986519 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:42.988715 (+ 2196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:42.988745 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:42.990747 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:42.993254 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.993260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.993280 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.993698 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.994090 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:42.994094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:42.994098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:42.994313 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:42.994931 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:42.994939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:42.995311 (+ 372us) client_negotiation.cc:769] Sending connection context
0812 11:13:42.996287 (+ 976us) client_negotiation.cc:240] Negotiation successful
0812 11:13:42.996693 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":179,"threads_started":1}
I20260812 11:13:42.999198 16033 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:42.999461 16033 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:43.000002 16033 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
W20260812 11:13:43.001776 16033 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:43517 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:13:43.001932 16033 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:43.002147 16033 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:43.003038 16033 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:43.003229 16033 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
W20260812 11:13:43.004011 16033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:13:43.125646 16039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:43.126056 16039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:43.126230 16039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:43.132131 16039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:43.132268 16039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:43.132417 16039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:13:43.139917 16039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.97.212:38149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:37179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=43615
--tserver_master_addrs=127.23.97.254:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:43.141554 16039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:43.142782 16039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:43.152812 16059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:43.153079 16039 server_base.cc:1034] running on GCE node
W20260812 11:13:43.152902 16061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:43.153323 16058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:43.153868 16039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:43.154647 16039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:43.155829 16039 hybrid_clock.cc:648] HybridClock initialized: now 1786533223155795 us; error 52 us; skew 500 ppm
Aug 12 11:13:43 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533223, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:43.159210 16039 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:13:43.160403 16039 webserver.cc:460] Webserver started at http://127.23.97.195:43615/ using document root <none> and password file <none>
I20260812 11:13:43.161197 16039 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:43.161316 16039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:13:43 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533223, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:43.188589 16039 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:13:43.191871 16077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:43.193404 16039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:13:43.193547 16039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d9273b27d5e54a21b9ee34768c4ae242"
format_stamp: "Formatted at 2026-08-12 11:13:41 on dist-test-slave-zpvx"
server_key: "507f250bd65b3f07a8da81556674f6ea"
server_key_iv: "2ad0428ff18a6b9d56e2f09a48030b30"
server_key_version: "kuduclusterkey@0"
I20260812 11:13:43.194134 16039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:43.232789 16039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:43.235889 16039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:43.236181 16039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:43.236948 16039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:43.237841 16039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:43.237914 16039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:43.237986 16039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:43.238039 16039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:43.249753 16039 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:37179
I20260812 11:13:43.249841 16315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:37179 every 8 connection(s)
I20260812 11:13:43.251101 16039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:13:43.260882 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 16039
Aug 12 11:13:43 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533223, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:43.268425 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:43.255129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:44437 (local address 127.23.97.254:43517)
0812 11:13:43.255285 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:43.255289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:43.255308 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:13:43.256276 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:43.256280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:43.256336 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:43.256464 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:43.257967 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.258787 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.260088 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.260511 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.260629 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:43.263882 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:43.263888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:43.263894 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:43.263928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:43.265823 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:43.266494 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:43.266500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:43.266505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:43.266608 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:43.267079 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:43.267082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:43.267084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:43.267346 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:43.267420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:43.268094 (+ 674us) server_negotiation.cc:299] Negotiation successful
0812 11:13:43.268260 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:13:43.269515 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:43.253942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43517 (local address 127.23.97.195:44437)
0812 11:13:43.254600 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:43.254662 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:43.256005 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:43.256653 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:43.256663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:43.257101 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:43.257775 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:43.257796 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.258943 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.258947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:43.259941 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:43.259951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.260714 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.260719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:43.260948 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:43.261747 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:43.261764 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:43.263681 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:43.265945 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:43.265951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:43.265968 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:43.266368 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:43.266729 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:43.266732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:43.266734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:43.266979 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:43.267459 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:43.267466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:43.267789 (+ 323us) client_negotiation.cc:769] Sending connection context
0812 11:13:43.268030 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 11:13:43.268422 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":176,"threads_started":1}
I20260812 11:13:43.270752 16316 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43517
I20260812 11:13:43.271014 16316 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:43.271627 16316 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:43.273274 15346 ts_manager.cc:194] Registered new tserver with Master: d9273b27d5e54a21b9ee34768c4ae242 (127.23.97.195:37179)
I20260812 11:13:43.274817 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:44437
I20260812 11:13:43.278187 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:43.261999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.23.97.254:43517)
0812 11:13:43.268677 (+ 6678us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:43.268681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:43.268693 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:13:43.268725 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:43.268728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:43.268766 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:43.268837 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:43.269892 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.270540 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.271549 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:43.271889 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:43.271985 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:43.273786 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:43.273790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:43.273792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:43.273819 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:43.275295 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:43.275717 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:43.275722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:43.275726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:43.275805 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:43.276110 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:43.276113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:43.276115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:43.276303 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:43.276507 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:43.277917 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0812 11:13:43.278024 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6623,"thread_start_us":134,"threads_started":1}
Aug 12 11:13:43 dist-test-slave-zpvx krb5kdc[13344](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533223, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:43.725863 15749 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:43.726213 15749 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:43.727044 15346 ts_manager.cc:194] Registered new tserver with Master: bc7974f191fe4748a7d22d2ee403ef55 (127.23.97.193:45729)
I20260812 11:13:43.727723 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:49771
I20260812 11:13:44.004285 16033 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:44.004635 16033 heartbeater.cc:511] Master 127.23.97.254:43517 requested a full tablet report, sending...
I20260812 11:13:44.005488 15346 ts_manager.cc:194] Registered new tserver with Master: fc29a9e899cb47e880067d1ff4136fa7 (127.23.97.194:43823)
I20260812 11:13:44.006122 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:45661
I20260812 11:13:44.011034 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:13:44.026360 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.014517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.23.97.254:43517)
0812 11:13:44.014842 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.014846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.017176 (+ 2330us) server_negotiation.cc:407] Connection header received
0812 11:13:44.017348 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.017351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.017408 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.017510 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:44.018266 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.018796 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.019631 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.019877 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.019988 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.021914 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:44.021919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:44.021922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:44.021950 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:44.024178 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:44.024815 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:44.024819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:44.024821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:44.024965 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:44.025343 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:44.025346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:44.025350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:44.025583 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:44.025742 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.026005 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.026150 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260812 11:13:44.030435 15346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:13:44.034925 15346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:13:44.053695 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.046442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43823 (local address 127.0.0.1:41810)
0812 11:13:44.048875 (+ 2433us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.048913 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:44.049087 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:44.049696 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:44.049702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:44.049727 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:44.049930 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.049940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.051188 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.051191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.052108 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.052120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.053114 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.053118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.053234 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.053273 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:13:44.053412 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:13:44.053539 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2253,"thread_start_us":156,"threads_started":1}
I20260812 11:13:44.053817 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.047504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.23.97.194:43823)
0812 11:13:44.048547 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.048556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.049019 (+ 463us) server_negotiation.cc:407] Connection header received
0812 11:13:44.049299 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.049308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.049535 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.049706 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:44.050038 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.051053 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.052245 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.052974 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.053101 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.053268 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.053444 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.053542 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":83,"threads_started":1}
I20260812 11:13:44.059082 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.046954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:37179 (local address 127.0.0.1:50206)
0812 11:13:44.053133 (+ 6179us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.053149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:44.053266 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:44.054065 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:44.054067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:44.054085 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:44.054192 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.054199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.055982 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.055986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.057212 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.057225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.058595 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.058599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.058693 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.058718 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:13:44.058793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 11:13:44.058863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6109,"thread_start_us":156,"threads_started":1}
I20260812 11:13:44.058568 15891 tablet_service.cc:1467] Processing CreateTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 (DEFAULT_TABLE table=test-table [id=3c94e6e19f3045cd80f610d38dec2d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:44.059574 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.047852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.23.97.195:37179)
0812 11:13:44.049437 (+ 1585us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.049443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.053367 (+ 3924us) server_negotiation.cc:407] Connection header received
0812 11:13:44.053449 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.053454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.053701 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.053879 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:44.054541 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.055848 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.057367 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.058276 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.058383 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.058524 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.059308 (+ 784us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.059431 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":137,"threads_started":1}
I20260812 11:13:44.060007 15891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b87d9df0d74db584e08e26c5ad5b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:44.060819 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.047372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:45729 (local address 127.0.0.1:37938)
0812 11:13:44.053709 (+ 6337us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.053723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:44.053843 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:44.055567 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:44.055570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:44.055590 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:44.055682 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.055689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.057219 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.057223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.058349 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.058362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.060420 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.060424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.060513 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.060541 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:13:44.060623 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:13:44.060691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6263,"thread_start_us":146,"threads_started":1}
I20260812 11:13:44.061342 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.047600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.23.97.193:45729)
0812 11:13:44.054490 (+ 6890us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.054497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.054521 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:13:44.054595 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.054600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.054769 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.054903 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:44.055826 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.057089 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.058504 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.059922 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.060037 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.060304 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.061050 (+ 746us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.061149 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6750,"thread_start_us":90,"threads_started":1}
I20260812 11:13:44.062487 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 (DEFAULT_TABLE table=test-table [id=3c94e6e19f3045cd80f610d38dec2d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:44.063484 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b87d9df0d74db584e08e26c5ad5b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:44.064404 15618 tablet_service.cc:1467] Processing CreateTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 (DEFAULT_TABLE table=test-table [id=3c94e6e19f3045cd80f610d38dec2d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:44.065550 15618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b87d9df0d74db584e08e26c5ad5b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:44.073369 16371 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Bootstrap starting.
I20260812 11:13:44.076094 16371 tablet_bootstrap.cc:654] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:44.077513 16371 log.cc:824] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:44.078392 16377 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Bootstrap starting.
I20260812 11:13:44.081527 16377 tablet_bootstrap.cc:654] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:44.082767 16377 log.cc:824] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:44.083830 16378 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Bootstrap starting.
I20260812 11:13:44.087497 16371 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: No bootstrap required, opened a new log
I20260812 11:13:44.087800 16371 ts_tablet_manager.cc:1397] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260812 11:13:44.094522 16378 tablet_bootstrap.cc:654] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:44.094910 16371 raft_consensus.cc:348] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.095310 16371 raft_consensus.cc:374] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:44.095388 16371 raft_consensus.cc:729] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9273b27d5e54a21b9ee34768c4ae242, State: Initialized, Role: FOLLOWER
I20260812 11:13:44.095705 16378 log.cc:824] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:44.095985 16371 consensus_queue.cc:260] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.099617 16377 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: No bootstrap required, opened a new log
I20260812 11:13:44.099924 16377 ts_tablet_manager.cc:1397] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260812 11:13:44.104560 16371 ts_tablet_manager.cc:1428] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260812 11:13:44.105037 16316 heartbeater.cc:503] Master 127.23.97.254:43517 was elected leader, sending a full tablet report...
I20260812 11:13:44.107472 16378 tablet_bootstrap.cc:492] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: No bootstrap required, opened a new log
I20260812 11:13:44.107811 16378 ts_tablet_manager.cc:1397] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260812 11:13:44.113008 16377 raft_consensus.cc:348] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.113353 16377 raft_consensus.cc:374] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:44.113440 16377 raft_consensus.cc:729] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7974f191fe4748a7d22d2ee403ef55, State: Initialized, Role: FOLLOWER
I20260812 11:13:44.114059 16377 consensus_queue.cc:260] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.114506 16378 raft_consensus.cc:348] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.114830 16378 raft_consensus.cc:374] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:44.114930 16378 raft_consensus.cc:729] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc29a9e899cb47e880067d1ff4136fa7, State: Initialized, Role: FOLLOWER
Aug 12 11:13:44 dist-test-slave-zpvx krb5kdc[13344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533223, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for HTTP/127.23.97.212@KRBTEST.COM
I20260812 11:13:44.115474 16378 consensus_queue.cc:260] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.115548 16377 ts_tablet_manager.cc:1428] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
I20260812 11:13:44.115979 15749 heartbeater.cc:503] Master 127.23.97.254:43517 was elected leader, sending a full tablet report...
I20260812 11:13:44.116761 16378 ts_tablet_manager.cc:1428] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260812 11:13:44.117141 16033 heartbeater.cc:503] Master 127.23.97.254:43517 was elected leader, sending a full tablet report...
W20260812 11:13:44.193847 15750 tablet.cc:2369] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:13:44.228345 16035 tablet.cc:2369] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:13:44.253302 16318 tablet.cc:2369] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:13:44.301718 16384 raft_consensus.cc:482] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:44.302067 16384 raft_consensus.cc:504] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.303511 16384 leader_election.cc:288] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc29a9e899cb47e880067d1ff4136fa7 (127.23.97.194:43823), d9273b27d5e54a21b9ee34768c4ae242 (127.23.97.195:37179)
I20260812 11:13:44.310207 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.304533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:46193 (local address 127.23.97.195:37179)
0812 11:13:44.304671 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.304676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.304934 (+ 258us) server_negotiation.cc:407] Connection header received
0812 11:13:44.305163 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.305166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.305212 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.305268 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:44.306013 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.306781 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.308605 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.309215 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.309347 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.309402 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.309965 (+ 563us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.310061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:13:44.310628 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.304409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:42417 (local address 127.23.97.194:43823)
0812 11:13:44.304568 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.304573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.304921 (+ 348us) server_negotiation.cc:407] Connection header received
0812 11:13:44.305206 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.305209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.305264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.305317 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:44.306393 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.307308 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.309179 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.309772 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.309881 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.309931 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.310397 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.310483 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:13:44.311224 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.304008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:37179 (local address 127.23.97.193:46193)
0812 11:13:44.304838 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.304852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:44.305042 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:44.305407 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:44.305410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:44.305426 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:44.305875 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.305884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.306952 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.306955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.308460 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.308472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.309505 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.309508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.309658 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.309683 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:13:44.310677 (+ 994us) client_negotiation.cc:240] Negotiation successful
0812 11:13:44.310828 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":6,"spinlock_wait_cycles":36096,"thread_start_us":177,"threads_started":1}
I20260812 11:13:44.311344 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.303809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:43823 (local address 127.23.97.193:42417)
0812 11:13:44.304808 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.304824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:44.305048 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:44.305472 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:44.305475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:44.305494 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:44.306225 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.306237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.307470 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.307473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.308941 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:44.308953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.310040 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.310044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:44.310227 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.310253 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:13:44.310954 (+ 701us) client_negotiation.cc:240] Negotiation successful
0812 11:13:44.311034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":213,"spinlock_wait_cycles":13696,"thread_start_us":193,"threads_started":1}
I20260812 11:13:44.312183 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b87d9df0d74db584e08e26c5ad5b11" candidate_uuid: "bc7974f191fe4748a7d22d2ee403ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9273b27d5e54a21b9ee34768c4ae242" is_pre_election: true
I20260812 11:13:44.312523 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b87d9df0d74db584e08e26c5ad5b11" candidate_uuid: "bc7974f191fe4748a7d22d2ee403ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc29a9e899cb47e880067d1ff4136fa7" is_pre_election: true
I20260812 11:13:44.312610 16215 raft_consensus.cc:2454] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7974f191fe4748a7d22d2ee403ef55 in term 0.
I20260812 11:13:44.312852 15893 raft_consensus.cc:2454] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7974f191fe4748a7d22d2ee403ef55 in term 0.
I20260812 11:13:44.313202 15519 leader_election.cc:302] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc7974f191fe4748a7d22d2ee403ef55, d9273b27d5e54a21b9ee34768c4ae242; no voters:
I20260812 11:13:44.313733 16384 raft_consensus.cc:2781] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:13:44.313841 16384 raft_consensus.cc:482] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:13:44.313892 16384 raft_consensus.cc:3037] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:44.315424 16384 raft_consensus.cc:504] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.315840 16384 leader_election.cc:288] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [CANDIDATE]: Term 1 election: Requested vote from peers fc29a9e899cb47e880067d1ff4136fa7 (127.23.97.194:43823), d9273b27d5e54a21b9ee34768c4ae242 (127.23.97.195:37179)
I20260812 11:13:44.316187 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b87d9df0d74db584e08e26c5ad5b11" candidate_uuid: "bc7974f191fe4748a7d22d2ee403ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc29a9e899cb47e880067d1ff4136fa7"
I20260812 11:13:44.316334 15893 raft_consensus.cc:3037] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:44.316437 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b87d9df0d74db584e08e26c5ad5b11" candidate_uuid: "bc7974f191fe4748a7d22d2ee403ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9273b27d5e54a21b9ee34768c4ae242"
I20260812 11:13:44.316569 16215 raft_consensus.cc:3037] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:44.318018 16215 raft_consensus.cc:2454] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7974f191fe4748a7d22d2ee403ef55 in term 1.
I20260812 11:13:44.318306 15893 raft_consensus.cc:2454] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7974f191fe4748a7d22d2ee403ef55 in term 1.
I20260812 11:13:44.318365 15519 leader_election.cc:302] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc7974f191fe4748a7d22d2ee403ef55, d9273b27d5e54a21b9ee34768c4ae242; no voters:
I20260812 11:13:44.318538 16384 raft_consensus.cc:2781] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:44.318797 16384 raft_consensus.cc:686] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [term 1 LEADER]: Becoming Leader. State: Replica: bc7974f191fe4748a7d22d2ee403ef55, State: Running, Role: LEADER
I20260812 11:13:44.319213 16384 consensus_queue.cc:237] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } }
I20260812 11:13:44.323241 15342 catalog_manager.cc:5495] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc7974f191fe4748a7d22d2ee403ef55 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "bc7974f191fe4748a7d22d2ee403ef55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc7974f191fe4748a7d22d2ee403ef55" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 45729 } health_report { overall_health: HEALTHY } } }
I20260812 11:13:44.352597 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.347463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.23.97.193:45729)
0812 11:13:44.347635 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:44.347640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:44.347657 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:13:44.347756 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:44.347759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:44.347831 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:44.347960 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:13:44.348274 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.349205 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.350494 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:44.350835 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:44.350951 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:44.351340 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:13:44.351962 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:13:44.352101 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:44.352239 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 11:13:44.352303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:13:44.360297 16215 raft_consensus.cc:1261] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 1 FOLLOWER]: Refusing update from remote peer bc7974f191fe4748a7d22d2ee403ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:44.360298 15893 raft_consensus.cc:1261] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 1 FOLLOWER]: Refusing update from remote peer bc7974f191fe4748a7d22d2ee403ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:44.361373 16397 consensus_queue.cc:1035] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc29a9e899cb47e880067d1ff4136fa7" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 43823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:44.361622 16384 consensus_queue.cc:1035] T 69b87d9df0d74db584e08e26c5ad5b11 P bc7974f191fe4748a7d22d2ee403ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9273b27d5e54a21b9ee34768c4ae242" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:44.374289 16405 mvcc.cc:205] Tried to move back new op lower bound from 7317640086970601472 to 7317640086815035392. Current Snapshot: MvccSnapshot[applied={T|T < 7317640086970601472}]
I20260812 11:13:44.379120 16411 mvcc.cc:205] Tried to move back new op lower bound from 7317640086970601472 to 7317640086815035392. Current Snapshot: MvccSnapshot[applied={T|T < 7317640086970601472 or (T in {7317640086970601472})}]
I20260812 11:13:44.387323 16407 mvcc.cc:205] Tried to move back new op lower bound from 7317640086970601472 to 7317640086815035392. Current Snapshot: MvccSnapshot[applied={T|T < 7317640086970601472}]
I20260812 11:13:44.388721 15342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:44.388911 15342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:44.391709 15342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1e99afa9bb944e3901826080dd194bd: Sending DeleteTablet for 3 replicas of tablet 69b87d9df0d74db584e08e26c5ad5b11
I20260812 11:13:44.392850 15891 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:44 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41810
I20260812 11:13:44.394457 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:44 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:50206
I20260812 11:13:44.395888 15618 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b87d9df0d74db584e08e26c5ad5b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:44 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:37938
I20260812 11:13:44.397758 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 15418
I20260812 11:13:44.398558 16425 tablet_replica.cc:330] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: stopping tablet replica
I20260812 11:13:44.398916 16425 raft_consensus.cc:2229] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:44.399262 16425 raft_consensus.cc:2258] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:44.399659 16427 tablet_replica.cc:330] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: stopping tablet replica
I20260812 11:13:44.399988 16427 raft_consensus.cc:2229] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:44.400245 16427 raft_consensus.cc:2258] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:44.402374 16425 ts_tablet_manager.cc:1905] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:44.407722 16427 ts_tablet_manager.cc:1905] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:44.412957 16425 ts_tablet_manager.cc:1918] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:44.413071 16425 log.cc:1196] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/69b87d9df0d74db584e08e26c5ad5b11
I20260812 11:13:44.413365 16427 ts_tablet_manager.cc:1918] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:44.413403 16425 ts_tablet_manager.cc:1939] T 69b87d9df0d74db584e08e26c5ad5b11 P fc29a9e899cb47e880067d1ff4136fa7: Deleting consensus metadata
I20260812 11:13:44.413465 16427 log.cc:1196] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/69b87d9df0d74db584e08e26c5ad5b11
I20260812 11:13:44.413813 16427 ts_tablet_manager.cc:1939] T 69b87d9df0d74db584e08e26c5ad5b11 P d9273b27d5e54a21b9ee34768c4ae242: Deleting consensus metadata
I20260812 11:13:44.414805 15312 catalog_manager.cc:4841] TS fc29a9e899cb47e880067d1ff4136fa7 (127.23.97.194:43823): tablet 69b87d9df0d74db584e08e26c5ad5b11 (table test-table [id=3c94e6e19f3045cd80f610d38dec2d41]) successfully deleted
I20260812 11:13:44.415747 15311 catalog_manager.cc:4841] TS d9273b27d5e54a21b9ee34768c4ae242 (127.23.97.195:37179): tablet 69b87d9df0d74db584e08e26c5ad5b11 (table test-table [id=3c94e6e19f3045cd80f610d38dec2d41]) successfully deleted
W20260812 11:13:44.422874 15311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:45729 (error 108)
I20260812 11:13:44.423076 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 15757
I20260812 11:13:44.427345 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:44.426952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:45729 (local address 127.0.0.1:37950)
0812 11:13:44.427165 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:44.427245 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260812 11:13:44.427735 15311 catalog_manager.cc:4584] TS bc7974f191fe4748a7d22d2ee403ef55 (127.23.97.193:45729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69b87d9df0d74db584e08e26c5ad5b11: Network error: Client connection negotiation failed: client connection to 127.23.97.193:45729: connect: Connection refused (error 111)
I20260812 11:13:44.433779 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 16039
I20260812 11:13:44.446013 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 15270
I20260812 11:13:44.467075 23943 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:13:44.815304 23943 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:13:44.815425 23943 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:13:45.170729 23943 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:13:45.170 UTC [13405] LOG: received smart shutdown request
2026-08-12 11:13:45.175 UTC [13405] LOG: background worker "logical replication launcher" (PID 13421) exited with exit code 1
2026-08-12 11:13:45.176 UTC [13412] LOG: shutting down
2026-08-12 11:13:45.329 UTC [13405] LOG: database system is shut down
2026-08-12T11:13:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48944 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:13:45 dist-test-slave-zpvx krb5kdc[16447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:13:45 dist-test-slave-zpvx krb5kdc[16447](info): set up 2 sockets
Aug 12 11:13:45 dist-test-slave-zpvx krb5kdc[16447](info): commencing operation
krb5kdc: starting...
W20260812 11:13:46.792284 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:13:46 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:13:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:13:46.933463 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33065
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:47.044780 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:47.045070 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:47.045148 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:47.049090 16469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:47.049172 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:47.049209 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:47.049268 16469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:47.049304 16469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:47.054381 16469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:47.055398 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:47.056416 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:47.064881 16481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:47.064858 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:47.064862 16476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:47.064913 16469 server_base.cc:1034] running on GCE node
I20260812 11:13:47.065687 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:47.066677 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:47.067838 16469 hybrid_clock.cc:648] HybridClock initialized: now 1786533227067812 us; error 40 us; skew 500 ppm
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:47.070879 16469 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:47.071844 16469 webserver.cc:460] Webserver started at http://127.23.97.254:33389/ using document root <none> and password file <none>
I20260812 11:13:47.072461 16469 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:47.072535 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:47.072724 16469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:47.075204 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "819eb30af01f48d789730097f88c5e07"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "0a95e43e2b4f2ef6fe885036fe98a51c"
server_key_iv: "ae805f23af56726f4932f40d63b641d6"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.075923 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "819eb30af01f48d789730097f88c5e07"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "0a95e43e2b4f2ef6fe885036fe98a51c"
server_key_iv: "ae805f23af56726f4932f40d63b641d6"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.080108 16469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:13:47.082631 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:47.083902 16469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:47.084057 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "819eb30af01f48d789730097f88c5e07"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "0a95e43e2b4f2ef6fe885036fe98a51c"
server_key_iv: "ae805f23af56726f4932f40d63b641d6"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.084168 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:47.095537 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:47.098704 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:47.098906 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:47.106223 16469 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:33065
I20260812 11:13:47.106253 16606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:33065 every 8 connection(s)
I20260812 11:13:47.107120 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:47.109910 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 16469
I20260812 11:13:47.110049 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:13:47.110332 23943 external_mini_cluster.cc:1428] Setting key 20bfce14016504dcd4a27a1cd4b28f36
I20260812 11:13:47.110332 16607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:47.114110 16607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.253@KRBTEST.COM" created.
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:47.129575 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.116584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:33065 (local address 127.0.0.1:44034)
0812 11:13:47.117160 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.117203 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.118011 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.118683 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.118688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.119112 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.119409 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.119422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.120698 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.120702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.121408 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.121417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.122330 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.122334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.122519 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.123114 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.123130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.124654 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.126599 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.126610 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.126619 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.126844 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.127180 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.127183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.127185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.127298 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.127885 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.127890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.128029 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.128212 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.128607 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":130,"spinlock_wait_cycles":13696,"thread_start_us":165,"threads_started":1}
I20260812 11:13:47.129576 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.116583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.23.97.254:33065)
0812 11:13:47.117007 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.117026 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.117315 (+ 289us) server_negotiation.cc:407] Connection header received
0812 11:13:47.118321 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.118325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.118498 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.118695 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.119557 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.120562 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.121544 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.122166 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.122335 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.124811 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.124816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.124825 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.124845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.126489 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.126955 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.126963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.126969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.127071 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.127393 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.127398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.127400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.127761 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.127902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.128278 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.128556 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":280,"thread_start_us":144,"threads_started":1}
I20260812 11:13:47.131847 16607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.132614 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.132260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43600)
0812 11:13:47.132465 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.132546 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 11:13:47.132851 16512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
I20260812 11:13:47.133677 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.133485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43608)
0812 11:13:47.133579 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.133624 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 11:13:47.133812 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36349: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
I20260812 11:13:47.138756 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:36349
--webserver_interface=127.23.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:47.183084 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } attempt: 1
I20260812 11:13:47.184010 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.183701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43616)
0812 11:13:47.183856 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.183930 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:13:47.184839 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.184592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43624)
0812 11:13:47.184724 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.184779 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:13:47.185003 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36349: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
W20260812 11:13:47.250905 16623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:47.251144 16623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:47.251184 16623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 11:13:47.253188 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } attempt: 2
I20260812 11:13:47.254043 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.253739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43638)
0812 11:13:47.253900 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.253968 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 11:13:47.254877 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.254654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43646)
0812 11:13:47.254771 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.254823 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 11:13:47.255041 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36349: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36349: connect: Connection refused (error 111)
W20260812 11:13:47.255376 16623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:47.255426 16623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:47.255445 16623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:47.255498 16623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:47.255537 16623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:47.260679 16623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:47.261713 16623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:47.262607 16623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:47.270889 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:47.270893 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:47.270905 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:47.271389 16623 server_base.cc:1034] running on GCE node
I20260812 11:13:47.272007 16623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:47.275206 16623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:47.276522 16623 hybrid_clock.cc:648] HybridClock initialized: now 1786533227276506 us; error 34 us; skew 500 ppm
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:47.279405 16623 init.cc:377] Logged in from keytab as kudu/127.23.97.253@KRBTEST.COM (short username kudu)
I20260812 11:13:47.280278 16623 webserver.cc:460] Webserver started at http://127.23.97.253:38163/ using document root <none> and password file <none>
I20260812 11:13:47.280882 16623 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:47.280947 16623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:47.281114 16623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:47.283241 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data/instance:
uuid: "bccfc1d9f16545a8aa98183af980db22"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "9c5c34627f54bc72db6e00e1d67c0016"
server_key_iv: "6bc1ac284c1d41a2cbc3b2bc56c633f9"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.283823 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal/instance:
uuid: "bccfc1d9f16545a8aa98183af980db22"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "9c5c34627f54bc72db6e00e1d67c0016"
server_key_iv: "6bc1ac284c1d41a2cbc3b2bc56c633f9"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.287662 16623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 11:13:47.290249 16659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:47.291519 16623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:47.291649 16623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "bccfc1d9f16545a8aa98183af980db22"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "9c5c34627f54bc72db6e00e1d67c0016"
server_key_iv: "6bc1ac284c1d41a2cbc3b2bc56c633f9"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.291741 16623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:47.306150 16623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:47.310055 16623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:47.310312 16623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:47.317893 16623 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.253:36349
I20260812 11:13:47.317902 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.253:36349 every 8 connection(s)
I20260812 11:13:47.318902 16623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
I20260812 11:13:47.321656 16771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:47.325323 16771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.326053 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 16623
I20260812 11:13:47.326195 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-1/wal/instance
I20260812 11:13:47.326495 23943 external_mini_cluster.cc:1428] Setting key b6761e48557e9658f1442acbfc562a3c
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.252@KRBTEST.COM" created.
I20260812 11:13:47.339447 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.327290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.23.97.254:33065)
0812 11:13:47.327427 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.327431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.328426 (+ 995us) server_negotiation.cc:407] Connection header received
0812 11:13:47.329825 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.329829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.329884 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.329961 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.331247 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.331781 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.332994 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.333236 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.333363 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.335835 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.335838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.335840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.335864 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.337232 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.337821 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.337826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.337828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.337913 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.338300 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.338303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.338305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.338522 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.338632 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.339133 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.339260 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:13:47.340343 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.327759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:33065 (local address 127.0.0.1:44048)
0812 11:13:47.328260 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.328302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.329569 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.330179 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.330192 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.330582 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.331093 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.331106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.331910 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.331913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.332869 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.332879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.333403 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.333408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.333580 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.334188 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.334205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.335658 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.337365 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.337371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.337385 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.337705 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.338009 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.338012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.338014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.338193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.338629 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.338635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.338885 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.339075 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.339390 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":157,"threads_started":1}
I20260812 11:13:47.341918 16771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.346211 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } attempt: 3
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:13:47.351590 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.342396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43656)
0812 11:13:47.342506 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.342522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.342596 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.343179 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.343182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.343350 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.343427 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.343433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.344389 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.344395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.344933 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.344941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.345523 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.345526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.345620 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.346080 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.346094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.347522 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.349749 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.349751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.349752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.349943 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.350281 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.350286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.350288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.350370 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.350930 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.350933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.351046 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.351193 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.351352 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 11:13:47.351908 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.342550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.23.97.253:36349)
0812 11:13:47.342863 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.342872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.342890 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:13:47.342921 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.342925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.343062 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.343202 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.343535 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.344244 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.345059 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.345410 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.345525 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.347675 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.347688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.347705 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.347734 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.349643 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.350089 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.350094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.350095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.350180 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.350474 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.350477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.350478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.350790 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.350937 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.351302 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.351427 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":139,"threads_started":1}
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:13:47.354166 16771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
I20260812 11:13:47.354887 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.354630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37230)
0812 11:13:47.354719 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.354798 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 11:13:47.355121 16664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
I20260812 11:13:47.355952 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:33719
--webserver_interface=127.23.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:47.356076 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.355820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37242)
0812 11:13:47.355958 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.356007 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 11:13:47.356208 16771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
I20260812 11:13:47.356453 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.346726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.23.97.253:36349)
0812 11:13:47.347054 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.347066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.347080 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:13:47.347113 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.347115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.347175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.347301 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.347776 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.348611 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.350031 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.350330 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.350441 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.352814 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.352817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.352819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.352840 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.354515 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.355109 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.355116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.355121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.355212 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.355533 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.355537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.355538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.355746 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.355867 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.356127 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.356244 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260812 11:13:47.356585 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.346669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43668)
0812 11:13:47.346781 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.346797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.346916 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.347334 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.347337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.347512 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.347653 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.347660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.348736 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.348740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.349635 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.349645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.350446 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.350450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.350571 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.351052 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.351066 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.352660 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.354659 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.354665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.354669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.354983 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.355311 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.355314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.355316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.355401 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.355858 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.355861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.355966 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.356076 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.356410 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 11:13:47.357250 16607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.357967 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.357705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37256)
0812 11:13:47.357835 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.357889 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 11:13:47.358850 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.358615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37258)
0812 11:13:47.358754 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.358806 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 11:13:47.358968 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
I20260812 11:13:47.390125 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 1
I20260812 11:13:47.391122 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.390759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37272)
0812 11:13:47.390946 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.391024 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 11:13:47.392050 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.391789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37282)
0812 11:13:47.391908 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.391968 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:13:47.392206 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
I20260812 11:13:47.405469 16771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 1
I20260812 11:13:47.406337 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.406032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37284)
0812 11:13:47.406200 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.406261 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:13:47.407056 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.406864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37298)
0812 11:13:47.406961 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.407004 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 11:13:47.407213 16771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
I20260812 11:13:47.467406 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 2
I20260812 11:13:47.468454 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.468068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37306)
0812 11:13:47.468257 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.468334 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 11:13:47.469304 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.469050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37312)
0812 11:13:47.469186 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.469241 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 11:13:47.469466 16607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
W20260812 11:13:47.473220 16794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:47.473502 16794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:47.473614 16794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 11:13:47.475430 16771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 2
I20260812 11:13:47.476166 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.475914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37328)
0812 11:13:47.476043 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.476096 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:13:47.476887 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.476700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37338)
0812 11:13:47.476797 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.476839 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 11:13:47.477013 16771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33719: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33719: connect: Connection refused (error 111)
W20260812 11:13:47.478223 16794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:47.478303 16794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:47.478346 16794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:47.478382 16794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:47.478417 16794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:47.483777 16794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:33065,127.23.97.253:36349,127.23.97.252:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:47.484978 16794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:47.486017 16794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:47.495229 16803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:47.495313 16794 server_base.cc:1034] running on GCE node
W20260812 11:13:47.495229 16805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:47.495536 16807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:47.496163 16794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:47.497357 16794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:47.498590 16794 hybrid_clock.cc:648] HybridClock initialized: now 1786533227498577 us; error 34 us; skew 500 ppm
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:47.501581 16794 init.cc:377] Logged in from keytab as kudu/127.23.97.252@KRBTEST.COM (short username kudu)
I20260812 11:13:47.502584 16794 webserver.cc:460] Webserver started at http://127.23.97.252:36177/ using document root <none> and password file <none>
I20260812 11:13:47.503273 16794 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:47.503342 16794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:47.503512 16794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:47.505657 16794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data/instance:
uuid: "75554f63c6144dc2bf204217b15f4c6d"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "262c544b4cd196161a017347d7abd85b"
server_key_iv: "4d0e379790ccdeda3eda913c1c48aee5"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.506181 16794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal/instance:
uuid: "75554f63c6144dc2bf204217b15f4c6d"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "262c544b4cd196161a017347d7abd85b"
server_key_iv: "4d0e379790ccdeda3eda913c1c48aee5"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.509961 16794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:13:47.512357 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:47.513690 16794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:47.513851 16794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "75554f63c6144dc2bf204217b15f4c6d"
format_stamp: "Formatted at 2026-08-12 11:13:47 on dist-test-slave-zpvx"
server_key: "262c544b4cd196161a017347d7abd85b"
server_key_iv: "4d0e379790ccdeda3eda913c1c48aee5"
server_key_version: "encryptionkey@0"
I20260812 11:13:47.513967 16794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:47.542404 16794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:47.545928 16794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:47.546157 16794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:47.553606 16794 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.252:33719
I20260812 11:13:47.553634 16912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.252:33719 every 8 connection(s)
I20260812 11:13:47.554585 16794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
I20260812 11:13:47.557436 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:47.561276 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.564102 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 16794
I20260812 11:13:47.564271 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533060973827-23943-0/minicluster-data/master-2/wal/instance
I20260812 11:13:47.564610 23943 external_mini_cluster.cc:1428] Setting key 0c067e6166fbbc3c302b596dfd81f271
I20260812 11:13:47.568164 16771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 3
I20260812 11:13:47.568625 16607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } attempt: 3
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
I20260812 11:13:47.577451 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.563151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.23.97.254:33065)
0812 11:13:47.563294 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.563298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.564195 (+ 897us) server_negotiation.cc:407] Connection header received
0812 11:13:47.565443 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.565446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.565507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.565600 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.566750 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.567382 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.569019 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.569376 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.569484 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.572544 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.572548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.572550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.572575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.574432 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.575144 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.575150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.575154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.575275 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.575779 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.575784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.575786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.576042 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.576144 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.577135 (+ 991us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.577268 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:13:47.578795 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.563498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:33065 (local address 127.0.0.1:44062)
0812 11:13:47.563994 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.564055 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.565210 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.565806 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.565813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.566141 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.566593 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.566605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.567492 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.567496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.568876 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.568889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.569654 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.569660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.569887 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.570572 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.570590 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.572413 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.574559 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.574565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.574579 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.575035 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.575447 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.575450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.575452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.575643 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.576215 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.576223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.576561 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.576767 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.577780 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":181,"thread_start_us":105,"threads_started":1}
I20260812 11:13:47.579911 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.568652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37350)
0812 11:13:47.568843 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.568857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.568941 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.569389 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.569395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.569593 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.569702 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.569710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.570772 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.570776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.571519 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.571530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.572225 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.572230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.572320 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.572841 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.572860 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.574424 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.578207 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.578210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.578212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.578468 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.578791 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.578794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.578796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.578894 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.579407 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.579410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.579516 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.579602 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.579764 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 11:13:47.580258 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.565364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.23.97.252:33719)
0812 11:13:47.565866 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.565880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.565912 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:13:47.565947 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.565950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.566103 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.566268 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.567116 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.568314 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.570670 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.571180 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.571254 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.575815 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.575820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.575825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.575857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.578192 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.578666 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.578669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.578671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.578785 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.579102 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.579106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.579107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.579317 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.579478 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.579902 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.580052 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":400,"thread_start_us":205,"threads_started":1}
I20260812 11:13:47.580523 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.568717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.23.97.252:33719)
0812 11:13:47.569127 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.569138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.569150 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:13:47.569185 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.569190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.569233 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.569815 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.569896 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.570649 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.571673 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.572069 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.573256 (+ 1187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.574626 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.574639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.574655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.574685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.578088 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.578587 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.578593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.578595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.578687 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.579010 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.579013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.579015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.579283 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.580078 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.580161 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.580296 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260812 11:13:47.580943 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.581646 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.569082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37356)
0812 11:13:47.569263 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.569278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.569351 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.569610 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.569613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.569806 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.569894 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.569900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.571006 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.571010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.571845 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.571859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.572522 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.572526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.572619 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.573094 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.573105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.575417 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.580001 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.580006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.580011 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.580249 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.580628 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.580631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.580633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.580728 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.581156 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.581159 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.581257 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.581368 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.581499 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 11:13:47.582201 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.569113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.23.97.252:33719)
0812 11:13:47.569363 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.569371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.569388 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:13:47.569424 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.569430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.569476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.570011 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.570128 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.570871 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.572007 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.572355 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.572830 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.575618 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.575623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.575625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.575650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.579885 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.580404 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.580408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.580410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.580499 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.580842 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.580844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.580846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.581021 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.581134 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.581882 (+ 748us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.582004 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260812 11:13:47.584954 16771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: Bootstrap starting.
I20260812 11:13:47.584959 16607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: Bootstrap starting.
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:13:47.587827 16771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:47.588011 16607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:47.588788 16771 log.cc:824] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:47.589131 16607 log.cc:824] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:47.591120 16771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: No bootstrap required, opened a new log
I20260812 11:13:47.591364 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.581482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.23.97.253:36349)
0812 11:13:47.582162 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.582166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.582313 (+ 147us) server_negotiation.cc:407] Connection header received
0812 11:13:47.582652 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.582658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.582715 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.582771 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.583289 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.584021 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.585084 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.585412 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.585489 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.587823 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.587830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.587836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.587866 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.589575 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.590079 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.590085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.590087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.590175 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.590470 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.590475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.590477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.590677 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.590788 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.591078 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.591210 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580}
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:47.591794 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.581412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36349 (local address 127.0.0.1:43680)
0812 11:13:47.582247 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.582261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.582369 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.582915 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.582917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.583067 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.583163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.583169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.584156 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.584159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.584937 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.584948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.585630 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.585634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.585741 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.586191 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.586207 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.587651 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.589716 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.589722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.589725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.589967 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.590281 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.590285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.590287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.590375 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.590780 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.590782 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.590879 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.591489 (+ 610us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.591639 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771}
I20260812 11:13:47.593444 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:13:47.593361 16607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: No bootstrap required, opened a new log
I20260812 11:13:47.595534 16771 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.595762 16771 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:47.595822 16771 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccfc1d9f16545a8aa98183af980db22, State: Initialized, Role: FOLLOWER
I20260812 11:13:47.595772 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.584631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.23.97.254:33065)
0812 11:13:47.584735 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.584739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.584898 (+ 159us) server_negotiation.cc:407] Connection header received
0812 11:13:47.585078 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.585081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.585122 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.585170 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.586302 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.587039 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.588479 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.589047 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.589152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.591978 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.591984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.591987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.592010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.593854 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.594313 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.594317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.594321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.594415 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.594728 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.594732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.594733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.594951 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.595152 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.595338 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.595522 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:13:47.596360 16771 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.597806 16771 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
I20260812 11:13:47.600404 16607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.600653 16607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:47.600720 16607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819eb30af01f48d789730097f88c5e07, State: Initialized, Role: FOLLOWER
I20260812 11:13:47.601195 16607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.602563 16607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:47.603164 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.593948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33719 (local address 127.0.0.1:37358)
0812 11:13:47.594085 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:47.594098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:47.594177 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:47.594467 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:47.594471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:47.594637 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:47.594743 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.594749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.595749 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.595752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.596628 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:47.596639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.597227 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.597231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:47.597310 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.597772 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:47.597787 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:47.599311 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:47.601471 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.601474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.601476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.601712 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.601982 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:47.601987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:47.601989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:47.602068 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:47.602528 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:47.602533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:47.602628 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 11:13:47.602816 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 11:13:47.602955 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:13:47.603822 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.593977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.23.97.252:33719)
0812 11:13:47.594069 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.594073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.594140 (+ 67us) server_negotiation.cc:407] Connection header received
0812 11:13:47.594257 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.594259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.594303 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.594370 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.594861 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.595634 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.596734 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.597032 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.597129 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.599453 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.599456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.599459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.599479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.601285 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.601811 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.601814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.601816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.601893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.602158 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.602161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.602162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.602389 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.602491 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.603507 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.603632 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Aug 12 11:13:47 dist-test-slave-zpvx krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:13:47.601423 16623 master_runner.cc:186] Error getting master registration for 127.23.97.254:33065: OK, instance_id { permanent_uuid: "819eb30af01f48d789730097f88c5e07" instance_seqno: 1786533227104081 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:13:47.605587 16623 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:13:47.606042 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.606249 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.606596 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.606732 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 11:13:47.608203 16969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:13:47.608317 16969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:13:47.611989 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: Bootstrap starting.
I20260812 11:13:47.612213 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.597377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.23.97.253:36349)
0812 11:13:47.598048 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.598052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.598065 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:13:47.598098 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.598101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.598139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.598221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.599080 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.600065 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.601471 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.601815 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.601908 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.607422 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.607426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.607429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.607453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.609852 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.610347 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.610351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.610353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.610448 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.610788 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.610794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.610799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.611015 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.611225 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.611866 (+ 641us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.611984 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588}
I20260812 11:13:47.610965 16469 master_runner.cc:186] Error getting master registration for 127.23.97.252:33719: OK, instance_id { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" instance_seqno: 1786533227551533 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:13:47.614887 16915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:47.614933 16469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:13:47.615772 16915 log.cc:824] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: Log is configured to *not* fsync() on all Append() calls
W20260812 11:13:47.616294 16991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:13:47.616411 16991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:13:47.617288 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:13:47.623883 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: No bootstrap required, opened a new log
I20260812 11:13:47.631263 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.619851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.23.97.253:36349)
0812 11:13:47.619965 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.619970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.621876 (+ 1906us) server_negotiation.cc:407] Connection header received
0812 11:13:47.622045 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.622048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.622095 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.622150 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.623046 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.623817 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.625221 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.625527 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.625597 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.627400 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.627404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.627406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.627431 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.629158 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.629684 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.629688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.629690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.629782 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.630135 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.630142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.630145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.630385 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.630514 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.630936 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.631060 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:13:47.635092 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.619408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.23.97.254:33065)
0812 11:13:47.619533 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.619537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.621540 (+ 2003us) server_negotiation.cc:407] Connection header received
0812 11:13:47.621669 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.621675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.621730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.621804 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.626415 (+ 4611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.627248 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.628969 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.629369 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.629455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.631101 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.631105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.631107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.631132 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.633169 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.633744 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.633748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.633749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.633835 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.634146 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.634150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.634151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.634354 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.634478 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.634826 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.634939 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:13:47.639259 16915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.639530 16915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:47.639606 16915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75554f63c6144dc2bf204217b15f4c6d, State: Initialized, Role: FOLLOWER
I20260812 11:13:47.640188 16915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.641044 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:47.621154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.23.97.252:33719)
0812 11:13:47.621311 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:47.621318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:47.628610 (+ 7292us) server_negotiation.cc:407] Connection header received
0812 11:13:47.628796 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:47.628799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:47.628844 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:47.628892 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:47.632730 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.633475 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.634590 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:47.634934 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:47.635007 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:47.637043 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:47.637048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:47.637052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:47.637079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:47.638926 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.639414 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.639417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.639419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.639507 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:47.639978 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:47.639982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:47.639983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:47.640190 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:47.640329 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:47.640805 (+ 476us) server_negotiation.cc:299] Negotiation successful
0812 11:13:47.640914 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 11:13:47.642066 16915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:13:47.648046 17034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:13:47.648181 17034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:13:47.652568 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.652710 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.653033 16794 master_runner.cc:263] No leader master found from master 75554f63c6144dc2bf204217b15f4c6d
I20260812 11:13:47.653110 16794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 11:13:47.797603 16825 tablet.cc:2369] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:13:47.811936 16672 tablet.cc:2369] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:13:47.850577 16518 tablet.cc:2369] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:13:47.922008 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:47.922255 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.923149 17014 leader_election.cc:288] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819eb30af01f48d789730097f88c5e07 (127.23.97.254:33065), bccfc1d9f16545a8aa98183af980db22 (127.23.97.253:36349)
I20260812 11:13:47.923779 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75554f63c6144dc2bf204217b15f4c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819eb30af01f48d789730097f88c5e07" is_pre_election: true
I20260812 11:13:47.923781 16711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75554f63c6144dc2bf204217b15f4c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccfc1d9f16545a8aa98183af980db22" is_pre_election: true
I20260812 11:13:47.924098 16561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75554f63c6144dc2bf204217b15f4c6d in term 0.
I20260812 11:13:47.924161 16711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75554f63c6144dc2bf204217b15f4c6d in term 0.
I20260812 11:13:47.924645 16821 leader_election.cc:302] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75554f63c6144dc2bf204217b15f4c6d, 819eb30af01f48d789730097f88c5e07; no voters:
I20260812 11:13:47.924927 17014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:13:47.925027 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:13:47.925086 17014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:47.926168 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.926486 17014 leader_election.cc:288] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [CANDIDATE]: Term 1 election: Requested vote from peers 819eb30af01f48d789730097f88c5e07 (127.23.97.254:33065), bccfc1d9f16545a8aa98183af980db22 (127.23.97.253:36349)
I20260812 11:13:47.926851 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75554f63c6144dc2bf204217b15f4c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819eb30af01f48d789730097f88c5e07"
I20260812 11:13:47.926914 16711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75554f63c6144dc2bf204217b15f4c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bccfc1d9f16545a8aa98183af980db22"
I20260812 11:13:47.927008 16561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:47.927043 16711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:47.928105 16711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75554f63c6144dc2bf204217b15f4c6d in term 1.
I20260812 11:13:47.928186 16561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75554f63c6144dc2bf204217b15f4c6d in term 1.
I20260812 11:13:47.928447 16821 leader_election.cc:302] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75554f63c6144dc2bf204217b15f4c6d, bccfc1d9f16545a8aa98183af980db22; no voters:
I20260812 11:13:47.928687 17014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:47.928918 17014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [term 1 LEADER]: Becoming Leader. State: Replica: 75554f63c6144dc2bf204217b15f4c6d, State: Running, Role: LEADER
I20260812 11:13:47.929250 17014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } }
I20260812 11:13:47.931039 17036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75554f63c6144dc2bf204217b15f4c6d. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.931162 17036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:47.931509 17038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:47.932358 17038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:47.936826 16561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [term 1 FOLLOWER]: Refusing update from remote peer 75554f63c6144dc2bf204217b15f4c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:47.936880 16711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [term 1 FOLLOWER]: Refusing update from remote peer 75554f63c6144dc2bf204217b15f4c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:47.937497 17014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:47.937664 17036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:47.941915 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75554f63c6144dc2bf204217b15f4c6d. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.942039 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.941967 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75554f63c6144dc2bf204217b15f4c6d. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.942091 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.944774 17046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.944794 17036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.944917 17046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:47.944924 17036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:47.945750 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.945859 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.946048 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75554f63c6144dc2bf204217b15f4c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819eb30af01f48d789730097f88c5e07" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33065 } } peers { permanent_uuid: "bccfc1d9f16545a8aa98183af980db22" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36349 } } peers { permanent_uuid: "75554f63c6144dc2bf204217b15f4c6d" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33719 } } }
I20260812 11:13:47.946161 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:13:47.946501 17038 catalog_manager.cc:1347] Generated new cluster ID: 7b15f9b9ad984b659c87c1f4066abf0f
I20260812 11:13:47.946570 17038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:47.959729 17038 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:13:47.960909 17038 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:47.970490 17038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75554f63c6144dc2bf204217b15f4c6d: Generated new TSK 0
I20260812 11:13:47.971285 17038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:13:48.609783 16969 catalog_manager.cc:1259] Loaded cluster ID: 7b15f9b9ad984b659c87c1f4066abf0f
I20260812 11:13:48.609874 16969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: loading cluster ID for follower catalog manager: success
I20260812 11:13:48.611501 16969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: acquiring CA information for follower catalog manager: success
I20260812 11:13:48.612653 16969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bccfc1d9f16545a8aa98183af980db22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:13:48.617695 16991 catalog_manager.cc:1259] Loaded cluster ID: 7b15f9b9ad984b659c87c1f4066abf0f
I20260812 11:13:48.617812 16991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: loading cluster ID for follower catalog manager: success
I20260812 11:13:48.619577 16991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: acquiring CA information for follower catalog manager: success
I20260812 11:13:48.620738 16991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 819eb30af01f48d789730097f88c5e07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:13:49.047658 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 16469
I20260812 11:13:49.062690 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 16623
I20260812 11:13:49.074662 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 16794
2026-08-12T11:13:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3628 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:13:49 dist-test-slave-zpvx krb5kdc[17065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:13:49 dist-test-slave-zpvx krb5kdc[17065](info): set up 2 sockets
Aug 12 11:13:49 dist-test-slave-zpvx krb5kdc[17065](info): commencing operation
krb5kdc: starting...
W20260812 11:13:50.380355 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:13:50.505461 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39647
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:50.621023 17087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:50.621301 17087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:50.621371 17087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:50.625458 17087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:50.625532 17087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:50.625568 17087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:50.625626 17087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:50.625666 17087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:50.630774 17087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:50.631762 17087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:50.632779 17087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:50.640921 17105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:50.640914 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:50.641084 17087 server_base.cc:1034] running on GCE node
W20260812 11:13:50.640976 17096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:50.641808 17087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:50.642820 17087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:50.643980 17087 hybrid_clock.cc:648] HybridClock initialized: now 1786533230643959 us; error 32 us; skew 500 ppm
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:50.646754 17087 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:50.647619 17087 webserver.cc:460] Webserver started at http://127.23.97.254:34025/ using document root <none> and password file <none>
I20260812 11:13:50.648156 17087 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:50.648248 17087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:50.648528 17087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:50.650476 17087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "a51c90216a214c1a849cf3fd324fe7a3"
format_stamp: "Formatted at 2026-08-12 11:13:50 on dist-test-slave-zpvx"
server_key: "fa4947ea94d077d6e2184cba5e2f3f8d"
server_key_iv: "621966608db8fe270c81d1737ce5c489"
server_key_version: "encryptionkey@0"
I20260812 11:13:50.651047 17087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "a51c90216a214c1a849cf3fd324fe7a3"
format_stamp: "Formatted at 2026-08-12 11:13:50 on dist-test-slave-zpvx"
server_key: "fa4947ea94d077d6e2184cba5e2f3f8d"
server_key_iv: "621966608db8fe270c81d1737ce5c489"
server_key_version: "encryptionkey@0"
I20260812 11:13:50.654646 17087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:13:50.656985 17115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:50.658186 17087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:13:50.658325 17087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "a51c90216a214c1a849cf3fd324fe7a3"
format_stamp: "Formatted at 2026-08-12 11:13:50 on dist-test-slave-zpvx"
server_key: "fa4947ea94d077d6e2184cba5e2f3f8d"
server_key_iv: "621966608db8fe270c81d1737ce5c489"
server_key_version: "encryptionkey@0"
I20260812 11:13:50.658434 17087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:50.674283 17087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:50.677412 17087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:50.677616 17087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:50.684545 17087 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:39647
I20260812 11:13:50.684573 17210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:39647 every 8 connection(s)
I20260812 11:13:50.689029 17087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:50.691491 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 17087
I20260812 11:13:50.691624 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:13:50.692337 23943 external_mini_cluster.cc:1428] Setting key d0636dc0befa5dfcc8326690740515a7
I20260812 11:13:50.692799 17212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:50.699959 17212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3: Bootstrap starting.
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:50.702644 17212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:50.703366 17212 log.cc:824] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:50.705498 17212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3: No bootstrap required, opened a new log
I20260812 11:13:50.707412 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:50.693731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.23.97.254:39647)
0812 11:13:50.694236 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:50.694254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:50.694419 (+ 165us) server_negotiation.cc:407] Connection header received
0812 11:13:50.695631 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:50.695649 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:50.695951 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:50.696218 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:50.697176 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:50.698131 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:50.699334 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:50.699799 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:50.699911 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:50.702725 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:50.702730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:50.702761 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:50.702796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:50.704764 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:50.705204 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:50.705210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:50.705217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:50.705354 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:50.705626 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:50.705635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:50.705637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:50.705982 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:50.706192 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:50.706500 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 11:13:50.706772 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
I20260812 11:13:50.709033 17212 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } }
I20260812 11:13:50.709324 17212 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:50.709396 17212 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51c90216a214c1a849cf3fd324fe7a3, State: Initialized, Role: FOLLOWER
I20260812 11:13:50.709786 17212 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } }
I20260812 11:13:50.709905 17212 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:50.710016 17212 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:50.710114 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:13:50.710173 17212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:50.711022 17212 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } }
I20260812 11:13:50.711387 17212 leader_election.cc:302] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51c90216a214c1a849cf3fd324fe7a3; no voters:
I20260812 11:13:50.711712 17212 leader_election.cc:288] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:13:50.711794 17219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:50.711977 17219 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [term 1 LEADER]: Becoming Leader. State: Replica: a51c90216a214c1a849cf3fd324fe7a3, State: Running, Role: LEADER
I20260812 11:13:50.712265 17219 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } }
I20260812 11:13:50.712950 17212 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:50.713967 17222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51c90216a214c1a849cf3fd324fe7a3. Latest consensus state: current_term: 1 leader_uuid: "a51c90216a214c1a849cf3fd324fe7a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } } }
I20260812 11:13:50.714133 17222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:50.714496 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51c90216a214c1a849cf3fd324fe7a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51c90216a214c1a849cf3fd324fe7a3" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 39647 } } }
I20260812 11:13:50.714570 17239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:50.714640 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:50.717782 17239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:50.722675 17239 catalog_manager.cc:1347] Generated new cluster ID: 8c540ea71dea495297cd5904158e5422
I20260812 11:13:50.722744 17239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:50.731530 17239 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:13:50.732411 17239 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:50.749745 17239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51c90216a214c1a849cf3fd324fe7a3: Generated new TSK 0
I20260812 11:13:50.750403 17239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:13:50.766438 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 17087
2026-08-12T11:13:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1693 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17261](info): set up 2 sockets
Aug 12 11:13:50 dist-test-slave-zpvx krb5kdc[17261](info): commencing operation
krb5kdc: starting...
W20260812 11:13:52.074272 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:13:52.197449 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:52.309046 17286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:52.309338 17286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:52.309407 17286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:52.313277 17286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:52.313357 17286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:52.313395 17286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:52.313453 17286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:52.313490 17286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:52.318738 17286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:52.319737 17286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:52.320761 17286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:52.330148 17296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.330209 17298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.330209 17302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:52.330603 17286 server_base.cc:1034] running on GCE node
I20260812 11:13:52.331149 17286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:52.332206 17286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:52.333371 17286 hybrid_clock.cc:648] HybridClock initialized: now 1786533232333349 us; error 35 us; skew 500 ppm
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:52.340283 17286 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:52.341151 17286 webserver.cc:460] Webserver started at http://127.23.97.254:38427/ using document root <none> and password file <none>
I20260812 11:13:52.341651 17286 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:52.341710 17286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:52.341869 17286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:52.343891 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "aab624656c9d4151a5057e12b30da147"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "8bc1afee982f8bb5805394b272f2a20f"
server_key_iv: "0ba20c06284294863f7983267270d5f4"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.344477 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "aab624656c9d4151a5057e12b30da147"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "8bc1afee982f8bb5805394b272f2a20f"
server_key_iv: "0ba20c06284294863f7983267270d5f4"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.347994 17286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:13:52.350478 17321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.351787 17286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:13:52.351956 17286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "aab624656c9d4151a5057e12b30da147"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "8bc1afee982f8bb5805394b272f2a20f"
server_key_iv: "0ba20c06284294863f7983267270d5f4"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.352077 17286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:52.369977 17286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:52.372782 17286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:52.372972 17286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:52.380043 17286 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34465
I20260812 11:13:52.380344 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34465 every 8 connection(s)
I20260812 11:13:52.381072 17286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:52.383689 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 17286
I20260812 11:13:52.383776 17406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:52.383857 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:13:52.384152 23943 external_mini_cluster.cc:1428] Setting key a1eb85c4b205a19faa79be9858d88825
I20260812 11:13:52.390350 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap starting.
I20260812 11:13:52.393287 17406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Neither blocks nor log segments found. Creating new log.
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:52.394037 17406 log.cc:824] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:52.396157 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: No bootstrap required, opened a new log
I20260812 11:13:52.398837 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:52.385506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.23.97.254:34465)
0812 11:13:52.385924 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:52.385940 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:52.385975 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:13:52.386800 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:52.386817 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:52.387126 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:52.387407 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:52.388300 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.389302 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.390487 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.390938 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.391050 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:52.393994 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:52.393999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:52.394013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:52.394040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:52.395964 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.396521 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.396526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.396531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.396638 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.396945 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.396949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.396950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.397374 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:52.397573 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:52.397852 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 11:13:52.398131 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":135,"threads_started":1}
I20260812 11:13:52.399816 17406 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:52.400049 17406 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:52.400128 17406 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab624656c9d4151a5057e12b30da147, State: Initialized, Role: FOLLOWER
I20260812 11:13:52.400596 17406 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:52.400758 17406 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:52.400831 17406 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:52.400952 17406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:52.401803 17406 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:52.402091 17406 leader_election.cc:302] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab624656c9d4151a5057e12b30da147; no voters:
I20260812 11:13:52.402333 17406 leader_election.cc:288] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:13:52.402467 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:52.402707 17424 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 LEADER]: Becoming Leader. State: Replica: aab624656c9d4151a5057e12b30da147, State: Running, Role: LEADER
I20260812 11:13:52.402992 17424 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:52.403506 17406 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:52.404654 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:52.404803 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:52.405164 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab624656c9d4151a5057e12b30da147. Latest consensus state: current_term: 1 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:52.405256 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:52.406337 17438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:52.409045 17438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:52.413997 17438 catalog_manager.cc:1347] Generated new cluster ID: 0039935664634b71a03257ca7ae856f3
I20260812 11:13:52.414060 17438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:52.422652 17438 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:13:52.423509 17438 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:52.427904 17438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Generated new TSK 0
I20260812 11:13:52.428653 17438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:13:52.464423 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:52.578634 17461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:52.578858 17461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:52.578898 17461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:52.582923 17461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:52.582978 17461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:52.583065 17461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:13:52.588055 17461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:52.589103 17461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:52.590095 17461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:52.599440 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.599470 17472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.599440 17470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:52.599644 17461 server_base.cc:1034] running on GCE node
I20260812 11:13:52.600170 17461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:52.600827 17461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:52.602020 17461 hybrid_clock.cc:648] HybridClock initialized: now 1786533232602012 us; error 30 us; skew 500 ppm
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:52.604815 17461 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:13:52.605780 17461 webserver.cc:460] Webserver started at http://127.23.97.193:36687/ using document root <none> and password file <none>
I20260812 11:13:52.606330 17461 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:52.606424 17461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:52.606650 17461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:52.608678 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "7d0b7ab15631216c0a037e6bbba7fd89"
server_key_iv: "2e4dd8a0e3304b2ba802a45b326d34b6"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.609266 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "7d0b7ab15631216c0a037e6bbba7fd89"
server_key_iv: "2e4dd8a0e3304b2ba802a45b326d34b6"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.613417 17461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 11:13:52.615795 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.617003 17461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:13:52.617153 17461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "7d0b7ab15631216c0a037e6bbba7fd89"
server_key_iv: "2e4dd8a0e3304b2ba802a45b326d34b6"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.617266 17461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:52.638368 17461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:52.641147 17461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:52.641350 17461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:52.641927 17461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:52.642742 17461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:52.642815 17461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.642886 17461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:52.642931 17461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.651979 17461 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44679
I20260812 11:13:52.651998 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44679 every 8 connection(s)
I20260812 11:13:52.652858 17461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:13:52.661468 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 17461
I20260812 11:13:52.661646 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:13:52.661964 23943 external_mini_cluster.cc:1428] Setting key 5721509b7c1b0b4620295441918dd7a3
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:52.666719 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:52.654673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:58745 (local address 127.23.97.254:34465)
0812 11:13:52.654803 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:52.654807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:52.655602 (+ 795us) server_negotiation.cc:407] Connection header received
0812 11:13:52.657069 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:52.657072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:52.657118 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:52.657192 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:52.658262 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.658789 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.659866 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.660078 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.660151 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:52.662800 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:52.662804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:52.662806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:52.662828 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:52.664593 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.665069 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.665076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.665096 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.665193 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.665511 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.665517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.665521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.665778 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:52.665893 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:52.666505 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 11:13:52.666604 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:13:52.668009 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:52.655023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:58745)
0812 11:13:52.655451 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:52.655494 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:52.656856 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:52.657325 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:52.657331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:52.657671 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:52.658135 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:52.658148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.658894 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.658896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:52.659760 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:52.659768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.660206 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.660209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:52.660341 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:52.660881 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:52.660897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:52.662639 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:52.664705 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:52.664710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:52.664721 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:52.664969 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:52.665292 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:52.665294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:52.665295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:52.665421 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:52.665908 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:52.665917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:52.666224 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 11:13:52.666464 (+ 240us) client_negotiation.cc:240] Negotiation successful
0812 11:13:52.666942 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":116,"threads_started":1}
I20260812 11:13:52.669291 17720 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:52.669534 17720 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:52.670008 17720 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:52.671499 17349 ts_manager.cc:194] Registered new tserver with Master: 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679)
I20260812 11:13:52.672703 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:58745
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:13:52.691308 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:52.804481 17730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:52.804711 17730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:52.804752 17730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:52.808959 17730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:52.809011 17730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:52.809083 17730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:13:52.814043 17730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:52.815034 17730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:52.815965 17730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:52.825687 17739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.825656 17738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:52.825659 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:52.825970 17730 server_base.cc:1034] running on GCE node
I20260812 11:13:52.826535 17730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:52.827139 17730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:52.828333 17730 hybrid_clock.cc:648] HybridClock initialized: now 1786533232828307 us; error 39 us; skew 500 ppm
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:52.831085 17730 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:13:52.831972 17730 webserver.cc:460] Webserver started at http://127.23.97.194:35753/ using document root <none> and password file <none>
I20260812 11:13:52.832589 17730 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:52.832681 17730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:52.832916 17730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:52.834869 17730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "7d344381ce274da3aa687b1589599884"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "13f56f5bb711cc9c2bacd4d66fc8dbff"
server_key_iv: "4d0b392fbc9ad0e388dc767b57ecc313"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.835430 17730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "7d344381ce274da3aa687b1589599884"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "13f56f5bb711cc9c2bacd4d66fc8dbff"
server_key_iv: "4d0b392fbc9ad0e388dc767b57ecc313"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.839107 17730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:13:52.841477 17758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.842608 17730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:13:52.842754 17730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "7d344381ce274da3aa687b1589599884"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "13f56f5bb711cc9c2bacd4d66fc8dbff"
server_key_iv: "4d0b392fbc9ad0e388dc767b57ecc313"
server_key_version: "encryptionkey@0"
I20260812 11:13:52.842862 17730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:52.861654 17730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:52.864902 17730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:52.865103 17730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:52.865684 17730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:52.866501 17730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:52.866573 17730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.866643 17730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:52.866698 17730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:52.875552 17730 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:36803
I20260812 11:13:52.875586 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:36803 every 8 connection(s)
I20260812 11:13:52.876473 17730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:13:52.878525 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 17730
I20260812 11:13:52.878696 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:13:52.878999 23943 external_mini_cluster.cc:1428] Setting key 39df45719d3be6b60186fefc45e2f1d5
Aug 12 11:13:52 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:13:52.891499 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:52.878274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:34301 (local address 127.23.97.254:34465)
0812 11:13:52.878435 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:52.878441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:52.879232 (+ 791us) server_negotiation.cc:407] Connection header received
0812 11:13:52.880626 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:52.880629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:52.880683 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:52.880772 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:52.881884 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.882722 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.884449 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.884868 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.884991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:52.887829 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:52.887832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:52.887834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:52.887857 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:52.889391 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.890041 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.890043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.890044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.890118 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:52.890518 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:52.890521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:52.890522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:52.890695 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:52.890780 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:52.891265 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 11:13:52.891378 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:13:52.892357 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:52.878615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:34301)
0812 11:13:52.879065 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:52.879117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:52.880429 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:52.880928 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:52.880934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:52.881273 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:52.881734 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:52.881746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.882876 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.882883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:52.884300 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:52.884310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:52.885023 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:52.885026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:52.885165 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:52.885793 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:52.885809 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:52.887647 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:52.889519 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:52.889527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:52.889548 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:52.889944 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:52.890208 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:52.890212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:52.890214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:52.890409 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:52.890808 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:52.890813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:52.891032 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 11:13:52.891199 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 11:13:52.891573 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:13:52.893553 17961 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:52.893801 17961 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:52.894284 17961 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:52.895375 17349 ts_manager.cc:194] Registered new tserver with Master: 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
I20260812 11:13:52.896212 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:34301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:13:52.909628 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:53.023670 17976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:53.023902 17976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:53.023943 17976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:53.027937 17976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:53.027992 17976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:53.028047 17976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:13:53.033097 17976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:53.033965 17976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:53.034866 17976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:53.044404 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:53.044409 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:53.044518 17976 server_base.cc:1034] running on GCE node
W20260812 11:13:53.044409 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:53.045065 17976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:53.045715 17976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:53.046880 17976 hybrid_clock.cc:648] HybridClock initialized: now 1786533233046864 us; error 27 us; skew 500 ppm
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:53.049925 17976 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:13:53.050769 17976 webserver.cc:460] Webserver started at http://127.23.97.195:37317/ using document root <none> and password file <none>
I20260812 11:13:53.051319 17976 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:53.051379 17976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:53.051587 17976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:13:53.053658 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "d4937befe7594f03bcd5c57988244868"
format_stamp: "Formatted at 2026-08-12 11:13:53 on dist-test-slave-zpvx"
server_key: "753ea356346f36b95ea997d9b6677799"
server_key_iv: "62cd9813af5b1a2835b0fb3805ed8233"
server_key_version: "encryptionkey@0"
I20260812 11:13:53.054208 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "d4937befe7594f03bcd5c57988244868"
format_stamp: "Formatted at 2026-08-12 11:13:53 on dist-test-slave-zpvx"
server_key: "753ea356346f36b95ea997d9b6677799"
server_key_iv: "62cd9813af5b1a2835b0fb3805ed8233"
server_key_version: "encryptionkey@0"
I20260812 11:13:53.057940 17976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:13:53.060243 18002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:53.061429 17976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:13:53.061550 17976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d4937befe7594f03bcd5c57988244868"
format_stamp: "Formatted at 2026-08-12 11:13:53 on dist-test-slave-zpvx"
server_key: "753ea356346f36b95ea997d9b6677799"
server_key_iv: "62cd9813af5b1a2835b0fb3805ed8233"
server_key_version: "encryptionkey@0"
I20260812 11:13:53.061652 17976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:53.079097 17976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:53.082335 17976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:53.082538 17976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:53.083107 17976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:53.083930 17976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:53.084002 17976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:53.084072 17976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:53.084122 17976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:53.093153 17976 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:35433
I20260812 11:13:53.093205 18209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:35433 every 8 connection(s)
I20260812 11:13:53.094038 17976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:13:53.096120 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 17976
I20260812 11:13:53.096278 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:13:53.096683 23943 external_mini_cluster.cc:1428] Setting key 5f14897c1e451c937483bdf39c4d5db3
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:53.108134 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.095829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:34047 (local address 127.23.97.254:34465)
0812 11:13:53.096016 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.096022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.096983 (+ 961us) server_negotiation.cc:407] Connection header received
0812 11:13:53.098703 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.098706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.098762 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.098882 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:53.100058 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.100604 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.101700 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.101910 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.101990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.104620 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:53.104623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:53.104625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:53.104649 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:53.106045 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.106569 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.106578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.106579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.106656 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.107004 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.107007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.107009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.107201 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:53.107312 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.107842 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.107962 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:13:53.109125 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.096192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:34047)
0812 11:13:53.096810 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.096868 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.098490 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.099083 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.099093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.099466 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:53.099932 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.099945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.100709 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.100712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.101582 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.101590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.102037 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.102040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.102173 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.102777 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:53.102792 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:53.104451 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:53.106176 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.106182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.106196 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.106454 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.106758 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.106762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.106764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.106904 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.107313 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:53.107321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:53.107607 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.107799 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.108111 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":169,"threads_started":1}
I20260812 11:13:53.110497 18210 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:53.110745 18210 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:53.111299 18210 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:53.112325 17349 ts_manager.cc:194] Registered new tserver with Master: d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433)
I20260812 11:13:53.112953 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:34047
I20260812 11:13:53.123037 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:13:53.132405 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.125140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.23.97.254:34465)
0812 11:13:53.125301 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.125307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.125470 (+ 163us) server_negotiation.cc:407] Connection header received
0812 11:13:53.125550 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.125553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.125601 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.125688 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:53.126409 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.126942 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.127832 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.128036 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.128146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.129528 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:53.129531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:53.129533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:53.129555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:53.130835 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.131229 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.131231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.131233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.131307 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.131537 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.131539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.131540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.131750 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:53.131853 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.132051 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.132141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:13:53.135262 17349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:13:53.138443 17349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:13:53.154047 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.148566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35433 (local address 127.0.0.1:59426)
0812 11:13:53.148800 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.148813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.148891 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.149790 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.149792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.149819 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:53.149924 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.149930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.151598 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.151601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.152721 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.152728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153554 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.153558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.153627 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.153645 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.153737 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.153860 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260812 11:13:53.154080 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.147550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.23.97.193:44679)
0812 11:13:53.148759 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.148776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.148801 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:13:53.148890 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.148896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.149132 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.149359 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:53.149701 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.150899 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.152432 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153294 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.153444 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.153606 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.153840 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.153949 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":136,"threads_started":1}
I20260812 11:13:53.154058 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.147372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44679 (local address 127.0.0.1:44512)
0812 11:13:53.147840 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.147907 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.148032 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.149371 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.149378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.149407 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:53.149557 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.149566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.151060 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.151066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.152285 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.152295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153450 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.153456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.153600 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.153646 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.153762 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.153860 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":6,"thread_start_us":89,"threads_started":1}
I20260812 11:13:53.154496 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.148134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36803 (local address 127.0.0.1:59388)
0812 11:13:53.148435 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.148450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.148535 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.150045 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.150048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.150067 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:53.150162 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.150169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.151153 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.151156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.152457 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.152468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153484 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.153488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.154142 (+ 654us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.154181 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.154265 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.154354 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":533,"thread_start_us":131,"threads_started":1}
I20260812 11:13:53.154901 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.148340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.23.97.194:36803)
0812 11:13:53.149571 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.149583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.149608 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:13:53.149691 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.149696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.149909 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.150049 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:53.150274 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.151009 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.152585 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153347 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.153577 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.153708 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.154667 (+ 959us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.154769 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":131,"threads_started":1}
I20260812 11:13:53.155457 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.148963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.23.97.195:35433)
0812 11:13:53.149343 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.149355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.149377 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:13:53.149452 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.149456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.149623 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.149808 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:53.150304 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.151442 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.152835 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.153422 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.155027 (+ 1605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.155179 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.155228 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.155318 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260812 11:13:53.157817 17588 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 (DEFAULT_TABLE table=test-table [id=e83d811a24de4441afe32eeff17b352e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:53.158136 17844 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 (DEFAULT_TABLE table=test-table [id=e83d811a24de4441afe32eeff17b352e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:53.158262 18096 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 (DEFAULT_TABLE table=test-table [id=e83d811a24de4441afe32eeff17b352e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:53.158959 17844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf5e65f7dc4f92978f55a4d169c111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:53.158993 17588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf5e65f7dc4f92978f55a4d169c111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:53.159198 18096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf5e65f7dc4f92978f55a4d169c111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:53.165566 18245 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Bootstrap starting.
I20260812 11:13:53.167119 18248 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Bootstrap starting.
I20260812 11:13:53.168736 18245 tablet_bootstrap.cc:654] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:53.168740 18246 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap starting.
I20260812 11:13:53.169740 18245 log.cc:824] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:53.169912 18248 tablet_bootstrap.cc:654] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:53.170771 18248 log.cc:824] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:53.170871 18246 tablet_bootstrap.cc:654] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:53.171722 18246 log.cc:824] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:53.172076 18245 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: No bootstrap required, opened a new log
I20260812 11:13:53.172288 18245 ts_tablet_manager.cc:1397] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:13:53.172807 18248 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: No bootstrap required, opened a new log
I20260812 11:13:53.172987 18248 ts_tablet_manager.cc:1397] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 11:13:53.173434 18246 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: No bootstrap required, opened a new log
I20260812 11:13:53.173646 18246 ts_tablet_manager.cc:1397] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:13:53.176309 18245 raft_consensus.cc:348] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.176613 18245 raft_consensus.cc:374] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:53.176697 18245 raft_consensus.cc:729] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d344381ce274da3aa687b1589599884, State: Initialized, Role: FOLLOWER
I20260812 11:13:53.176847 18248 raft_consensus.cc:348] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.177090 18248 raft_consensus.cc:374] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:53.177135 18248 raft_consensus.cc:729] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4937befe7594f03bcd5c57988244868, State: Initialized, Role: FOLLOWER
I20260812 11:13:53.176970 18246 raft_consensus.cc:348] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.177232 18246 raft_consensus.cc:374] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:53.177170 18245 consensus_queue.cc:260] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.177289 18246 raft_consensus.cc:729] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140966be3c0d48c6ba3cdc559fa1fab1, State: Initialized, Role: FOLLOWER
I20260812 11:13:53.177605 18248 consensus_queue.cc:260] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.177723 18246 consensus_queue.cc:260] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.178175 18245 ts_tablet_manager.cc:1428] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 11:13:53.178514 18248 ts_tablet_manager.cc:1428] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 11:13:53.178558 17961 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
I20260812 11:13:53.178663 18246 ts_tablet_manager.cc:1428] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260812 11:13:53.178866 18210 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
I20260812 11:13:53.178964 17720 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
I20260812 11:13:53.284396 18258 raft_consensus.cc:482] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:53.284693 18258 raft_consensus.cc:504] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.286000 18258 leader_election.cc:288] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803), d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433)
I20260812 11:13:53.290625 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.286981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:38545 (local address 127.23.97.195:35433)
0812 11:13:53.287113 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.287117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.287220 (+ 103us) server_negotiation.cc:407] Connection header received
0812 11:13:53.287397 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.287400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.287450 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.287536 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:53.287806 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.288298 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.289643 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.290042 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.290184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.290240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.290425 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.290497 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:13:53.290670 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.286784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35433 (local address 127.23.97.193:38545)
0812 11:13:53.287118 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.287133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.287291 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.287549 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.287552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.287568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:53.287696 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.287701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.288447 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.288454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.289479 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.289490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.290184 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.290188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.290287 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.290317 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.290408 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.290476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"spinlock_wait_cycles":4864,"thread_start_us":113,"threads_started":1}
I20260812 11:13:53.291196 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.286278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36803 (local address 127.23.97.193:39183)
0812 11:13:53.286643 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.286661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.286785 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.287222 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.287226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.287284 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:53.287406 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.287411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.288188 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.288192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.289566 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.289581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.290654 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.290661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.290786 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.290823 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.290893 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.290976 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":197,"spinlock_wait_cycles":50432,"thread_start_us":148,"threads_started":1}
I20260812 11:13:53.291412 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.286425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:39183 (local address 127.23.97.194:36803)
0812 11:13:53.286573 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.286577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.286766 (+ 189us) server_negotiation.cc:407] Connection header received
0812 11:13:53.286989 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.286992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.287057 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.287185 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:53.287536 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.288065 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.289730 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.290382 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.290512 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.290586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.291175 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.291240 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:13:53.291528 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf5e65f7dc4f92978f55a4d169c111" candidate_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4937befe7594f03bcd5c57988244868" is_pre_election: true
I20260812 11:13:53.291908 18125 raft_consensus.cc:2454] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140966be3c0d48c6ba3cdc559fa1fab1 in term 0.
I20260812 11:13:53.292002 17879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf5e65f7dc4f92978f55a4d169c111" candidate_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d344381ce274da3aa687b1589599884" is_pre_election: true
I20260812 11:13:53.292315 17879 raft_consensus.cc:2454] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140966be3c0d48c6ba3cdc559fa1fab1 in term 0.
I20260812 11:13:53.292510 17494 leader_election.cc:302] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1, d4937befe7594f03bcd5c57988244868; no voters:
I20260812 11:13:53.292799 18258 raft_consensus.cc:2781] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:13:53.292907 18258 raft_consensus.cc:482] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:13:53.292966 18258 raft_consensus.cc:3037] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:53.294026 18258 raft_consensus.cc:504] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.294373 18258 leader_election.cc:288] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803), d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433)
I20260812 11:13:53.294799 17879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf5e65f7dc4f92978f55a4d169c111" candidate_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d344381ce274da3aa687b1589599884"
I20260812 11:13:53.294818 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf5e65f7dc4f92978f55a4d169c111" candidate_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4937befe7594f03bcd5c57988244868"
I20260812 11:13:53.294898 17879 raft_consensus.cc:3037] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:53.294945 18125 raft_consensus.cc:3037] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:53.295989 18125 raft_consensus.cc:2454] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140966be3c0d48c6ba3cdc559fa1fab1 in term 1.
I20260812 11:13:53.295997 17879 raft_consensus.cc:2454] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140966be3c0d48c6ba3cdc559fa1fab1 in term 1.
I20260812 11:13:53.296267 17494 leader_election.cc:302] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1, d4937befe7594f03bcd5c57988244868; no voters:
I20260812 11:13:53.296468 18258 raft_consensus.cc:2781] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:53.296691 18258 raft_consensus.cc:686] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 LEADER]: Becoming Leader. State: Replica: 140966be3c0d48c6ba3cdc559fa1fab1, State: Running, Role: LEADER
I20260812 11:13:53.297026 18258 consensus_queue.cc:237] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.300442 17348 catalog_manager.cc:5495] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } health_report { overall_health: UNKNOWN } } }
I20260812 11:13:53.343467 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.340114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.23.97.193:44679)
0812 11:13:53.340247 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.340251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.340288 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:13:53.340404 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.340408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.340485 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.340576 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:13:53.340792 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.341296 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.342098 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.342319 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.342395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.342733 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:13:53.343129 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:13:53.343218 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.343310 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.343358 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 11:13:53.345031 18213 tablet.cc:2369] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:13:53.349740 18125 raft_consensus.cc:1261] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Refusing update from remote peer 140966be3c0d48c6ba3cdc559fa1fab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:53.349740 17879 raft_consensus.cc:1261] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Refusing update from remote peer 140966be3c0d48c6ba3cdc559fa1fab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:53.350523 18265 consensus_queue.cc:1035] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:53.350764 18258 consensus_queue.cc:1035] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:53.358484 18269 mvcc.cc:205] Tried to move back new op lower bound from 7317640123792793600 to 7317640123587567616. Current Snapshot: MvccSnapshot[applied={T|T < 7317640123792793600}]
I20260812 11:13:53.359611 18271 mvcc.cc:205] Tried to move back new op lower bound from 7317640123792793600 to 7317640123587567616. Current Snapshot: MvccSnapshot[applied={T|T < 7317640123792793600}]
I20260812 11:13:53.360078 18272 mvcc.cc:205] Tried to move back new op lower bound from 7317640123792793600 to 7317640123587567616. Current Snapshot: MvccSnapshot[applied={T|T < 7317640123792793600}]
I20260812 11:13:53.368155 17347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:53.368338 17347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:53.370761 17347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Sending DeleteTablet for 3 replicas of tablet 04cf5e65f7dc4f92978f55a4d169c111
I20260812 11:13:53.371629 18096 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:53 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59426
I20260812 11:13:53.371825 17588 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:53 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:44512
I20260812 11:13:53.371850 17844 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:53 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59388
I20260812 11:13:53.372311 18289 tablet_replica.cc:330] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: stopping tablet replica
I20260812 11:13:53.372327 18287 tablet_replica.cc:330] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: stopping tablet replica
I20260812 11:13:53.372463 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 17461
I20260812 11:13:53.372644 18287 raft_consensus.cc:2229] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:53.372644 18289 raft_consensus.cc:2229] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:13:53.378644 18290 tablet_replica.cc:330] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: stopping tablet replica
I20260812 11:13:53.378935 18290 raft_consensus.cc:2229] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:53.379197 18290 raft_consensus.cc:2258] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:53.380883 18287 raft_consensus.cc:2258] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:53.384096 18287 ts_tablet_manager.cc:1905] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:53.386909 18287 ts_tablet_manager.cc:1918] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:53.387001 18287 log.cc:1196] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/04cf5e65f7dc4f92978f55a4d169c111
I20260812 11:13:53.387312 18287 ts_tablet_manager.cc:1939] T 04cf5e65f7dc4f92978f55a4d169c111 P d4937befe7594f03bcd5c57988244868: Deleting consensus metadata
I20260812 11:13:53.389020 17325 catalog_manager.cc:4841] TS d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): tablet 04cf5e65f7dc4f92978f55a4d169c111 (table test-table [id=e83d811a24de4441afe32eeff17b352e]) successfully deleted
I20260812 11:13:53.393311 18290 ts_tablet_manager.cc:1905] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:53.395591 18290 ts_tablet_manager.cc:1918] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:53.395709 18290 log.cc:1196] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/04cf5e65f7dc4f92978f55a4d169c111
I20260812 11:13:53.396041 18290 ts_tablet_manager.cc:1939] T 04cf5e65f7dc4f92978f55a4d169c111 P 7d344381ce274da3aa687b1589599884: Deleting consensus metadata
W20260812 11:13:53.396587 17325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44679 (error 108)
I20260812 11:13:53.396858 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 17730
I20260812 11:13:53.400338 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.399975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44679 (local address 127.0.0.1:44540)
0812 11:13:53.400168 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.400241 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 11:13:53.401186 17325 catalog_manager.cc:4584] TS 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04cf5e65f7dc4f92978f55a4d169c111: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44679: connect: Connection refused (error 111)
I20260812 11:13:53.407063 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 17976
I20260812 11:13:53.436689 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.436295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36803 (local address 127.0.0.1:59396)
0812 11:13:53.436524 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.436593 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.194:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 11:13:53.442072 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 17286
I20260812 11:13:53.460258 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--webserver_interface=127.23.97.254
--webserver_port=38427
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:53.573674 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:53.573925 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:53.574000 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:53.577932 18293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:53.577986 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:53.578004 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:53.578063 18293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:53.578087 18293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:53.583262 18293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:53.584297 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:53.585315 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:53.599311 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:53.599308 18304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:53.599308 18303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:53.599618 18293 server_base.cc:1034] running on GCE node
I20260812 11:13:53.600359 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:53.601534 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:53.602723 18293 hybrid_clock.cc:648] HybridClock initialized: now 1786533233602702 us; error 39 us; skew 500 ppm
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:53.605512 18293 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:53.606462 18293 webserver.cc:460] Webserver started at http://127.23.97.254:38427/ using document root <none> and password file <none>
I20260812 11:13:53.607049 18293 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:53.607174 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:53.611308 18293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260812 11:13:53.613936 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:53.615098 18293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:13:53.615280 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "aab624656c9d4151a5057e12b30da147"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "8bc1afee982f8bb5805394b272f2a20f"
server_key_iv: "0ba20c06284294863f7983267270d5f4"
server_key_version: "encryptionkey@0"
I20260812 11:13:53.615891 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:53.640898 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:53.643685 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:53.643899 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:53.651095 18293 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34465
I20260812 11:13:53.651404 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34465 every 8 connection(s)
I20260812 11:13:53.652272 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:53.656720 18404 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:13:53.657310 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 18293
I20260812 11:13:53.657827 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:44679
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=36687
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:53.658906 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap starting.
I20260812 11:13:53.676311 18404 log.cc:824] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:53.685412 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:13:53.685799 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap complete.
I20260812 11:13:53.689957 18404 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:53.690333 18404 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab624656c9d4151a5057e12b30da147, State: Initialized, Role: FOLLOWER
I20260812 11:13:53.690861 18404 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:53.690980 18404 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:53.691057 18404 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:53.691180 18404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:13:53.693183 18404 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:53.693521 18404 leader_election.cc:302] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab624656c9d4151a5057e12b30da147; no voters:
I20260812 11:13:53.693877 18404 leader_election.cc:288] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:13:53.693958 18415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:13:53.694175 18415 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 LEADER]: Becoming Leader. State: Replica: aab624656c9d4151a5057e12b30da147, State: Running, Role: LEADER
I20260812 11:13:53.694456 18415 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:53.695101 18404 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:53.695446 18419 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab624656c9d4151a5057e12b30da147. Latest consensus state: current_term: 2 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:53.695586 18419 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:53.695814 18418 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:53.695920 18418 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:53.695971 18421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:53.700790 18421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:53.701419 18421 catalog_manager.cc:1259] Loaded cluster ID: 0039935664634b71a03257ca7ae856f3
I20260812 11:13:53.701507 18421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:13:53.702971 18421 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 11:13:53.707659 18421 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 11:13:53.712996 18421 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:13:53.713778 18421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Loaded TSK: 0
I20260812 11:13:53.714923 18421 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:13:53.775472 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:53.775745 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:53.775820 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:53.779827 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:53.779909 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:53.779994 18410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:13:53.785120 18410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:44679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=36687
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:53.786083 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:53.787058 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:53.796530 18463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:53.796597 18458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:53.796748 18457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:53.797091 18410 server_base.cc:1034] running on GCE node
I20260812 11:13:53.797552 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:53.798307 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:53.799515 18410 hybrid_clock.cc:648] HybridClock initialized: now 1786533233799491 us; error 39 us; skew 500 ppm
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:53.803026 18410 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:13:53.804177 18410 webserver.cc:460] Webserver started at http://127.23.97.193:36687/ using document root <none> and password file <none>
I20260812 11:13:53.804947 18410 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:53.805063 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:53.810125 18410 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:13:53.812800 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:53.814352 18410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:13:53.814476 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "7d0b7ab15631216c0a037e6bbba7fd89"
server_key_iv: "2e4dd8a0e3304b2ba802a45b326d34b6"
server_key_version: "encryptionkey@0"
I20260812 11:13:53.815157 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:53.838943 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:53.843012 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:53.843220 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:53.843873 18410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:53.845139 18484 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:13:53.849321 18410 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:13:53.849392 18410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260812 11:13:53.849450 18410 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:13:53.851511 18410 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:13:53.851568 18410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 11:13:53.851966 18484 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap starting.
I20260812 11:13:53.862619 18410 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44679
I20260812 11:13:53.862666 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44679 every 8 connection(s)
I20260812 11:13:53.863552 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:13:53.865501 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 18410
I20260812 11:13:53.865983 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:36803
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=35753
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:53.871630 18484 log.cc:824] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:53.881253 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.865944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:41575)
0812 11:13:53.867208 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.867267 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.868274 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.869455 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.869463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.869921 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:53.870378 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.870391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.871711 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.871715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.872920 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.872931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.873671 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.873677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.873872 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.874511 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:53.874528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:53.875992 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:53.878169 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.878176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.878188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.878473 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.878848 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.878851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.878853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.878992 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.879570 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:53.879586 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:53.879817 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.880006 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.880333 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":173,"threads_started":1}
I20260812 11:13:53.881242 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.865990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:41575 (local address 127.23.97.254:34465)
0812 11:13:53.866911 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.866931 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.867414 (+ 483us) server_negotiation.cc:407] Connection header received
0812 11:13:53.868749 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.868760 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.869046 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.869399 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:53.870562 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.871524 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.873063 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.873514 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.873717 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.876197 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:53.876202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:53.876215 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:53.876242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:53.878030 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.878597 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.878610 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.878615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.878735 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.879113 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.879121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.879122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.879450 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:53.879598 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.880132 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.880428 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":155,"threads_started":1}
I20260812 11:13:53.881687 18484 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:13:53.882175 18484 tablet_bootstrap.cc:492] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap complete.
I20260812 11:13:53.883749 18484 ts_tablet_manager.cc:1397] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20260812 11:13:53.883781 18657 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:53.884022 18657 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:53.884548 18657 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:53.886773 18343 ts_manager.cc:194] Registered new tserver with Master: 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679)
I20260812 11:13:53.886644 18484 raft_consensus.cc:348] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.887157 18484 raft_consensus.cc:729] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140966be3c0d48c6ba3cdc559fa1fab1, State: Initialized, Role: FOLLOWER
I20260812 11:13:53.887638 18484 consensus_queue.cc:260] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } }
I20260812 11:13:53.888643 18484 ts_tablet_manager.cc:1428] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260812 11:13:53.890527 18343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:41575
I20260812 11:13:53.893010 18657 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
Aug 12 11:13:53 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533233, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:13:53.901106 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.889897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.23.97.193:44679)
0812 11:13:53.890162 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:53.890173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:53.890869 (+ 696us) server_negotiation.cc:407] Connection header received
0812 11:13:53.891051 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:53.891056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:53.891253 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:53.891421 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:53.892171 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.893198 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.894517 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.894949 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.895035 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.897533 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:53.897541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:53.897551 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:53.897578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:53.899167 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.899672 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.899678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.899680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.899764 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:53.900072 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:53.900075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:53.900078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:53.900309 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:53.900477 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:53.900814 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 11:13:53.900970 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
I20260812 11:13:53.901104 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:53.889848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44679 (local address 127.0.0.1:44546)
0812 11:13:53.890718 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:53.890764 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:53.890904 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:53.891460 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:53.891466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:53.891904 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:53.892032 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.892040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.893321 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.893324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.894313 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:53.894326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:53.895068 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:53.895072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:53.895187 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:53.895794 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:53.895809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:53.897389 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:53.899302 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.899311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.899317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.899550 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.899885 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:53.899888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:53.899890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:53.899979 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:53.900451 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:53.900457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:53.900586 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 11:13:53.900728 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 11:13:53.900917 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":127,"threads_started":1}
I20260812 11:13:53.902107 18553 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf5e65f7dc4f92978f55a4d169c111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:53 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:44546
I20260812 11:13:53.902981 18675 tablet_replica.cc:330] stopping tablet replica
I20260812 11:13:53.903292 18675 raft_consensus.cc:2229] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:53.903537 18675 raft_consensus.cc:2258] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:53.904651 18675 ts_tablet_manager.cc:1905] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:53.909134 18675 ts_tablet_manager.cc:1918] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:53.909240 18675 log.cc:1196] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/04cf5e65f7dc4f92978f55a4d169c111
I20260812 11:13:53.909561 18675 ts_tablet_manager.cc:1939] T 04cf5e65f7dc4f92978f55a4d169c111 P 140966be3c0d48c6ba3cdc559fa1fab1: Deleting consensus metadata
I20260812 11:13:53.911491 18322 catalog_manager.cc:4841] TS 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679): tablet 04cf5e65f7dc4f92978f55a4d169c111 (table test-table [id=e83d811a24de4441afe32eeff17b352e]) successfully deleted
W20260812 11:13:53.990908 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:53.991158 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:53.991199 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:53.995529 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:53.995589 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:53.995683 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:13:54.001735 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:36803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=35753
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:54.002789 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:54.003798 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:54.013624 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.013624 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.013672 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:54.014045 18667 server_base.cc:1034] running on GCE node
I20260812 11:13:54.014434 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:54.015054 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:54.016219 18667 hybrid_clock.cc:648] HybridClock initialized: now 1786533234016175 us; error 54 us; skew 500 ppm
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:54.018970 18667 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:13:54.019897 18667 webserver.cc:460] Webserver started at http://127.23.97.194:35753/ using document root <none> and password file <none>
I20260812 11:13:54.020533 18667 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:54.020638 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:54.024410 18667 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:13:54.026402 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.027624 18667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:54.027765 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "7d344381ce274da3aa687b1589599884"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "13f56f5bb711cc9c2bacd4d66fc8dbff"
server_key_iv: "4d0b392fbc9ad0e388dc767b57ecc313"
server_key_version: "encryptionkey@0"
I20260812 11:13:54.028297 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:54.046507 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:54.049598 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:54.049839 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:54.050431 18667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:54.051251 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:54.051299 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.051365 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:54.051402 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.060632 18667 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:36803
I20260812 11:13:54.060652 18907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:36803 every 8 connection(s)
I20260812 11:13:54.061601 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:13:54.066663 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 18667
I20260812 11:13:54.067142 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:35433
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=37317
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:54.077010 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.063478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:33647 (local address 127.23.97.254:34465)
0812 11:13:54.063622 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.063626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.064517 (+ 891us) server_negotiation.cc:407] Connection header received
0812 11:13:54.065913 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.065916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.065981 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.066090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:54.067415 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.068266 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.069892 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.070134 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.070224 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.072856 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:54.072863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:54.072869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:54.072906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:54.074698 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.075199 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.075205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.075209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.075316 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.075651 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.075657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.075660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.075930 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:54.076050 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.076580 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.076763 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:13:54.077554 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.063863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:33647)
0812 11:13:54.064310 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.064351 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.065688 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.066273 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.066280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.066631 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:54.067153 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.067166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.068474 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.068477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.069758 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.069772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.070314 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.070319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.070533 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.071166 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:54.071182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:54.072675 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:54.074819 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:54.074824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:54.074834 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:54.075080 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:54.075414 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:54.075416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:54.075417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:54.075538 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:54.076040 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:54.076045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:54.076259 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.076476 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.076725 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260812 11:13:54.078794 18908 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:54.079041 18908 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:54.079524 18908 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:54.080582 18343 ts_manager.cc:194] Registered new tserver with Master: 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
I20260812 11:13:54.081128 18343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:33647
W20260812 11:13:54.181576 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:54.181805 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:54.181846 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:54.185797 18912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:54.185845 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:54.185901 18912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:13:54.190864 18912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:35433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=37317
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:54.191788 18912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:54.192780 18912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:54.202397 18922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.202397 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.202543 18917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:54.203130 18912 server_base.cc:1034] running on GCE node
I20260812 11:13:54.203512 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:54.204171 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:54.205334 18912 hybrid_clock.cc:648] HybridClock initialized: now 1786533234205296 us; error 49 us; skew 500 ppm
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:54.207949 18912 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:13:54.208896 18912 webserver.cc:460] Webserver started at http://127.23.97.195:37317/ using document root <none> and password file <none>
I20260812 11:13:54.209460 18912 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:54.209533 18912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:54.213613 18912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:13:54.215672 18934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.216887 18912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:13:54.217018 18912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d4937befe7594f03bcd5c57988244868"
format_stamp: "Formatted at 2026-08-12 11:13:53 on dist-test-slave-zpvx"
server_key: "753ea356346f36b95ea997d9b6677799"
server_key_iv: "62cd9813af5b1a2835b0fb3805ed8233"
server_key_version: "encryptionkey@0"
I20260812 11:13:54.217563 18912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:54.231909 18912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:54.234687 18912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:54.234876 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:54.235440 18912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:54.236241 18912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:54.236311 18912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.236405 18912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:54.236443 18912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.245469 18912 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:35433
I20260812 11:13:54.245491 19154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:35433 every 8 connection(s)
I20260812 11:13:54.246426 18912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:13:54.254864 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 18912
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:13:54.260226 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.248231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:37343 (local address 127.23.97.254:34465)
0812 11:13:54.248424 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.248429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.249323 (+ 894us) server_negotiation.cc:407] Connection header received
0812 11:13:54.250592 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.250595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.250643 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.250721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:54.251817 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.252332 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.253508 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.253726 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.253813 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.256671 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:54.256675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:54.256677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:54.256699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:54.258125 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.258660 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.258664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.258665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.258750 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.259095 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.259098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.259100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.259356 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:54.259460 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.259937 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.260055 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:13:54.261037 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.248676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:37343)
0812 11:13:54.249141 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.249182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.250371 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.250883 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.250889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.251222 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:54.251679 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.251693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.252508 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.252511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.253371 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.253380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.253889 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.253892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.254039 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.254649 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:54.254672 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:54.256501 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:54.258261 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:54.258274 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:54.258288 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:54.258538 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:54.258850 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:54.258854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:54.258856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:54.258996 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:54.259466 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:54.259475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:54.259706 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.259891 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.260190 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
I20260812 11:13:54.262274 19156 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34465
I20260812 11:13:54.262521 19156 heartbeater.cc:464] Registering TS with master...
I20260812 11:13:54.263098 19156 heartbeater.cc:511] Master 127.23.97.254:34465 requested a full tablet report, sending...
I20260812 11:13:54.263880 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.255572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.23.97.254:34465)
0812 11:13:54.255830 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.255834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.256030 (+ 196us) server_negotiation.cc:407] Connection header received
0812 11:13:54.256216 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.256218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.256258 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.256356 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:54.257125 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.257628 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.258529 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.258759 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.258872 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.260199 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:54.260202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:54.260204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:54.260226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:54.262113 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.262631 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.262636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.262638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.262735 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:54.263021 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:54.263023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:54.263024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:54.263240 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:54.263394 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.263586 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.263723 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260812 11:13:54.264344 18343 ts_manager.cc:194] Registered new tserver with Master: d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433)
I20260812 11:13:54.265091 18343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:37343
I20260812 11:13:54.265687 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:13:54.269610 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.266323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.23.97.254:34465)
0812 11:13:54.266445 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.266449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.266617 (+ 168us) server_negotiation.cc:407] Connection header received
0812 11:13:54.266783 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.266785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.266864 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.266943 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:13:54.267169 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.267615 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.268441 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.268644 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.268719 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.268967 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:13:54.269284 (+ 317us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:13:54.269366 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.269423 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.269466 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:13:54.270165 18341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:13:54.273228 18341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:13:54.280807 18553 tablet_service.cc:1467] Processing CreateTablet for tablet ea3412500e354d649ce2b7bcbddb3267 (DEFAULT_TABLE table=test-table [id=cda644dd65024ac2adc2541fd6e598c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:54.281446 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3412500e354d649ce2b7bcbddb3267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:54.284296 18484 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap starting.
I20260812 11:13:54.284487 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.279476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35433 (local address 127.0.0.1:59440)
0812 11:13:54.280037 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.280050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.280161 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.280624 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.280627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.280643 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:54.280725 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.280731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.282150 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.282153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.283194 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.283203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.284050 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.284054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.284142 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.284189 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.284275 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.284359 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488}
I20260812 11:13:54.284574 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.279661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.23.97.195:35433)
0812 11:13:54.279991 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.280003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.280172 (+ 169us) server_negotiation.cc:407] Connection header received
0812 11:13:54.280263 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.280268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.280490 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.280653 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:54.280841 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.281988 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.283323 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.283916 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.284024 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.284171 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.284313 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.284436 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":159,"threads_started":1}
I20260812 11:13:54.285198 18484 tablet_bootstrap.cc:654] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:54.285288 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.279827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36803 (local address 127.0.0.1:59408)
0812 11:13:54.280361 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.280396 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.280485 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.281011 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.281014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.281033 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:54.281112 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.281117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.282494 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.282501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.283738 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.283750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.284904 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.284908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.284993 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.285014 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.285103 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.285171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":102,"threads_started":1}
I20260812 11:13:54.285360 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.280002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.23.97.194:36803)
0812 11:13:54.280508 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.280519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.280564 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:13:54.280655 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.280661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.280856 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.281114 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:54.281276 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.282352 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.283902 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.284771 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.284901 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.285032 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.285130 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.285226 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":98,"threads_started":1}
I20260812 11:13:54.287046 18484 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: No bootstrap required, opened a new log
I20260812 11:13:54.287143 18484 ts_tablet_manager.cc:1397] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:13:54.287572 18484 raft_consensus.cc:348] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.287694 18484 raft_consensus.cc:374] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:54.287739 18484 raft_consensus.cc:729] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140966be3c0d48c6ba3cdc559fa1fab1, State: Initialized, Role: FOLLOWER
I20260812 11:13:54.287873 18484 consensus_queue.cc:260] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.288130 18484 ts_tablet_manager.cc:1428] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 11:13:54.288266 19026 tablet_service.cc:1467] Processing CreateTablet for tablet ea3412500e354d649ce2b7bcbddb3267 (DEFAULT_TABLE table=test-table [id=cda644dd65024ac2adc2541fd6e598c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:54.288868 18785 tablet_service.cc:1467] Processing CreateTablet for tablet ea3412500e354d649ce2b7bcbddb3267 (DEFAULT_TABLE table=test-table [id=cda644dd65024ac2adc2541fd6e598c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:13:54.289464 19026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3412500e354d649ce2b7bcbddb3267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:54.289872 18785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3412500e354d649ce2b7bcbddb3267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:13:54.296440 19176 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Bootstrap starting.
I20260812 11:13:54.296496 19177 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Bootstrap starting.
I20260812 11:13:54.298825 19176 tablet_bootstrap.cc:654] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:54.299480 19177 tablet_bootstrap.cc:654] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Neither blocks nor log segments found. Creating new log.
I20260812 11:13:54.299681 19176 log.cc:824] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:54.300208 19177 log.cc:824] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:54.301541 19176 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: No bootstrap required, opened a new log
I20260812 11:13:54.301743 19176 ts_tablet_manager.cc:1397] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:13:54.301954 19177 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: No bootstrap required, opened a new log
I20260812 11:13:54.302143 19177 ts_tablet_manager.cc:1397] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:13:54.304730 19176 raft_consensus.cc:348] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.305018 19176 raft_consensus.cc:374] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:54.305104 19176 raft_consensus.cc:729] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4937befe7594f03bcd5c57988244868, State: Initialized, Role: FOLLOWER
I20260812 11:13:54.305061 19177 raft_consensus.cc:348] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.305342 19177 raft_consensus.cc:374] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:13:54.305400 19177 raft_consensus.cc:729] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d344381ce274da3aa687b1589599884, State: Initialized, Role: FOLLOWER
I20260812 11:13:54.305589 19176 consensus_queue.cc:260] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.305830 19177 consensus_queue.cc:260] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.306236 19156 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
I20260812 11:13:54.306437 18908 heartbeater.cc:503] Master 127.23.97.254:34465 was elected leader, sending a full tablet report...
I20260812 11:13:54.306555 19176 ts_tablet_manager.cc:1428] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 11:13:54.306766 19177 ts_tablet_manager.cc:1428] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260812 11:13:54.312582 18910 tablet.cc:2369] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:13:54.362372 19185 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:54.362619 19185 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.363849 19185 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679)
W20260812 11:13:54.364923 18662 tablet.cc:2369] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:13:54.367698 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.364207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:41353 (local address 127.23.97.195:35433)
0812 11:13:54.364350 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.364356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.364397 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:13:54.364569 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.364575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.364640 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.364714 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:54.365135 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.365728 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.366743 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.367136 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.367279 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.367336 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.367509 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.367578 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:13:54.367698 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.364096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35433 (local address 127.23.97.194:41353)
0812 11:13:54.364234 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.364252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.364360 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.364850 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.364853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.364873 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:54.364993 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.365003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.365849 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.365854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.366614 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.366623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.367262 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.367265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.367376 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.367403 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.367493 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.367566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 11:13:54.368122 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.364437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44679 (local address 127.23.97.194:59799)
0812 11:13:54.364847 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.364860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.364991 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.365361 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:54.365363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:54.365376 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:13:54.365464 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.365468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.366257 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.366263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.367026 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:54.367035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.367746 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.367749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:54.367835 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.367855 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 11:13:54.367922 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 11:13:54.367984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":153,"threads_started":1}
I20260812 11:13:54.368117 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.364621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:59799 (local address 127.23.97.193:44679)
0812 11:13:54.364774 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.364778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.364949 (+ 171us) server_negotiation.cc:407] Connection header received
0812 11:13:54.365128 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.365136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.365191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.365248 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:13:54.365593 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.366132 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.367170 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.367600 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.367745 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.367886 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.367936 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.368003 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:13:54.368414 19063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3412500e354d649ce2b7bcbddb3267" candidate_uuid: "7d344381ce274da3aa687b1589599884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4937befe7594f03bcd5c57988244868" is_pre_election: true
I20260812 11:13:54.368733 19063 raft_consensus.cc:2454] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d344381ce274da3aa687b1589599884 in term 0.
I20260812 11:13:54.369150 18595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3412500e354d649ce2b7bcbddb3267" candidate_uuid: "7d344381ce274da3aa687b1589599884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" is_pre_election: true
I20260812 11:13:54.369283 18706 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868; no voters:
I20260812 11:13:54.369447 18595 raft_consensus.cc:2454] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d344381ce274da3aa687b1589599884 in term 0.
I20260812 11:13:54.369604 19185 raft_consensus.cc:2781] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:13:54.369720 19185 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:13:54.369792 19185 raft_consensus.cc:3037] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:54.370886 19185 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.371212 19185 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [CANDIDATE]: Term 1 election: Requested vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679)
I20260812 11:13:54.371531 19063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3412500e354d649ce2b7bcbddb3267" candidate_uuid: "7d344381ce274da3aa687b1589599884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4937befe7594f03bcd5c57988244868"
I20260812 11:13:54.371615 18595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3412500e354d649ce2b7bcbddb3267" candidate_uuid: "7d344381ce274da3aa687b1589599884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
I20260812 11:13:54.371686 19063 raft_consensus.cc:3037] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:54.371721 18595 raft_consensus.cc:3037] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:13:54.373052 19063 raft_consensus.cc:2454] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d344381ce274da3aa687b1589599884 in term 1.
I20260812 11:13:54.373052 18595 raft_consensus.cc:2454] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d344381ce274da3aa687b1589599884 in term 1.
I20260812 11:13:54.373353 18706 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868; no voters:
I20260812 11:13:54.373561 19185 raft_consensus.cc:2781] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:13:54.373787 19185 raft_consensus.cc:686] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 1 LEADER]: Becoming Leader. State: Replica: 7d344381ce274da3aa687b1589599884, State: Running, Role: LEADER
I20260812 11:13:54.374112 19185 consensus_queue.cc:237] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.377427 18343 catalog_manager.cc:5495] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d344381ce274da3aa687b1589599884 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "7d344381ce274da3aa687b1589599884" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } health_report { overall_health: UNKNOWN } } }
I20260812 11:13:54.410540 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.406812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.23.97.194:36803)
0812 11:13:54.406950 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:54.406956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:54.406972 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:13:54.407093 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:54.407095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:54.407188 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:54.407310 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:13:54.407557 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.408077 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.409029 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:54.409263 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:54.409373 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:54.409723 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:13:54.410139 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:13:54.410276 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:54.410339 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 11:13:54.410395 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:13:54.417101 18595 raft_consensus.cc:1261] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Refusing update from remote peer 7d344381ce274da3aa687b1589599884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:54.417104 19063 raft_consensus.cc:1261] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Refusing update from remote peer 7d344381ce274da3aa687b1589599884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:13:54.417847 19185 consensus_queue.cc:1035] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:54.418089 19191 consensus_queue.cc:1035] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:13:54.424835 19200 mvcc.cc:205] Tried to move back new op lower bound from 7317640128163667968 to 7317640127999152128. Current Snapshot: MvccSnapshot[applied={T|T < 7317640128163667968}]
I20260812 11:13:54.425741 19202 mvcc.cc:205] Tried to move back new op lower bound from 7317640128163667968 to 7317640127999152128. Current Snapshot: MvccSnapshot[applied={T|T < 7317640128163667968}]
I20260812 11:13:54.432933 18341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:54.433073 18341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:13:54.435343 18341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Sending DeleteTablet for 3 replicas of tablet ea3412500e354d649ce2b7bcbddb3267
I20260812 11:13:54.436048 18553 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3412500e354d649ce2b7bcbddb3267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:54 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:44546
I20260812 11:13:54.436113 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 18410
I20260812 11:13:54.436069 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3412500e354d649ce2b7bcbddb3267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:54 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59440
I20260812 11:13:54.444547 18785 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3412500e354d649ce2b7bcbddb3267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:13:54 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:59408
I20260812 11:13:54.446393 19217 tablet_replica.cc:330] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: stopping tablet replica
I20260812 11:13:54.446676 19217 raft_consensus.cc:2229] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:13:54.447016 19217 raft_consensus.cc:2258] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:54.447399 19214 tablet_replica.cc:330] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: stopping tablet replica
I20260812 11:13:54.447695 19214 raft_consensus.cc:2229] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:13:54.448006 19214 raft_consensus.cc:2258] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:13:54.449527 19214 ts_tablet_manager.cc:1905] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:54.449791 19217 ts_tablet_manager.cc:1905] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:13:54.451779 19214 ts_tablet_manager.cc:1918] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:54.451910 19214 log.cc:1196] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/ea3412500e354d649ce2b7bcbddb3267
I20260812 11:13:54.452246 19214 ts_tablet_manager.cc:1939] T ea3412500e354d649ce2b7bcbddb3267 P d4937befe7594f03bcd5c57988244868: Deleting consensus metadata
I20260812 11:13:54.453225 19217 ts_tablet_manager.cc:1918] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:13:54.453337 19217 log.cc:1196] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/ea3412500e354d649ce2b7bcbddb3267
I20260812 11:13:54.453430 18322 catalog_manager.cc:4841] TS d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): tablet ea3412500e354d649ce2b7bcbddb3267 (table test-table [id=cda644dd65024ac2adc2541fd6e598c8]) successfully deleted
I20260812 11:13:54.453680 19217 ts_tablet_manager.cc:1939] T ea3412500e354d649ce2b7bcbddb3267 P 7d344381ce274da3aa687b1589599884: Deleting consensus metadata
W20260812 11:13:54.454597 18322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44679 (error 108)
I20260812 11:13:54.454892 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 18667
I20260812 11:13:54.455150 18322 catalog_manager.cc:4841] TS 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803): tablet ea3412500e354d649ce2b7bcbddb3267 (table test-table [id=cda644dd65024ac2adc2541fd6e598c8]) successfully deleted
I20260812 11:13:54.455829 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.455444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44679 (local address 127.0.0.1:44548)
0812 11:13:54.455615 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.455715 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:13:54.457901 18322 catalog_manager.cc:4584] TS 140966be3c0d48c6ba3cdc559fa1fab1 (127.23.97.193:44679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea3412500e354d649ce2b7bcbddb3267: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44679: connect: Connection refused (error 111)
I20260812 11:13:54.469157 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 18912
I20260812 11:13:54.484236 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 18293
I20260812 11:13:54.493193 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--webserver_interface=127.23.97.254
--webserver_port=38427
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:54.603845 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:54.604099 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:54.604138 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:54.608129 19219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:13:54.608188 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:54.608207 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:54.608224 19219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:13:54.608238 19219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:13:54.613591 19219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:54.614660 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:54.615703 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:54.624199 19232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.624195 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.624228 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:54.624691 19219 server_base.cc:1034] running on GCE node
I20260812 11:13:54.625239 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:54.626310 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:54.627482 19219 hybrid_clock.cc:648] HybridClock initialized: now 1786533234627444 us; error 46 us; skew 500 ppm
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:54.630339 19219 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:13:54.631206 19219 webserver.cc:460] Webserver started at http://127.23.97.254:38427/ using document root <none> and password file <none>
I20260812 11:13:54.631757 19219 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:54.631820 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:54.635526 19219 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:54.637691 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.638828 19219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:13:54.638965 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "aab624656c9d4151a5057e12b30da147"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "8bc1afee982f8bb5805394b272f2a20f"
server_key_iv: "0ba20c06284294863f7983267270d5f4"
server_key_version: "encryptionkey@0"
I20260812 11:13:54.639492 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:54.667448 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:54.670387 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:54.670598 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:54.677534 19219 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34465
I20260812 11:13:54.677845 19341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34465 every 8 connection(s)
I20260812 11:13:54.678627 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:13:54.679899 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 19219
I20260812 11:13:54.680439 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:44679
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=36687
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:54.683748 19343 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:13:54.685760 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap starting.
I20260812 11:13:54.706979 19343 log.cc:824] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:54.718231 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:13:54.718614 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147: Bootstrap complete.
I20260812 11:13:54.722658 19343 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:54.723073 19343 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab624656c9d4151a5057e12b30da147, State: Initialized, Role: FOLLOWER
I20260812 11:13:54.723592 19343 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:54.723702 19343 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:13:54.723791 19343 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:13:54.723910 19343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:13:54.725754 19343 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:54.726082 19343 leader_election.cc:302] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab624656c9d4151a5057e12b30da147; no voters:
I20260812 11:13:54.726413 19343 leader_election.cc:288] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:13:54.726493 19347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:13:54.726707 19347 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [term 3 LEADER]: Becoming Leader. State: Replica: aab624656c9d4151a5057e12b30da147, State: Running, Role: LEADER
I20260812 11:13:54.727000 19347 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } }
I20260812 11:13:54.727615 19343 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:13:54.727948 19350 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab624656c9d4151a5057e12b30da147. Latest consensus state: current_term: 3 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:54.728083 19350 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:54.728307 19349 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aab624656c9d4151a5057e12b30da147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab624656c9d4151a5057e12b30da147" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34465 } } }
I20260812 11:13:54.728428 19349 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab624656c9d4151a5057e12b30da147 [sys.catalog]: This master's current role is: LEADER
I20260812 11:13:54.728492 19354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:13:54.733639 19354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:13:54.734342 19354 catalog_manager.cc:1259] Loaded cluster ID: 0039935664634b71a03257ca7ae856f3
I20260812 11:13:54.734406 19354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 11:13:54.735495 19354 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9e670be9ad google::LogMessage::Fail() at ??:0
@ 0x7f9e670c188a google::LogMessage::SendToLog() at ??:0
@ 0x7f9e670be4de google::LogMessage::Flush() at ??:0
@ 0x7f9e670bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9e719fdfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9e719fed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9e719fb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9e71a3f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9e693103d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9e69310c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9e693123f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9e69301512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9e6a9456db start_thread at ??:0
@ 0x7f9e65bbc61f clone at ??:0
W20260812 11:13:54.831624 19345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:54.831867 19345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:54.831910 19345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:54.836000 19345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:54.836061 19345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:54.836117 19345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:13:54.841723 19345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:44679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=36687
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:54.842772 19345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:54.843815 19345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:54.853591 19415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.853619 19420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:54.853806 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:54.854066 19345 server_base.cc:1034] running on GCE node
I20260812 11:13:54.854451 19345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:54.855096 19345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:54.856308 19345 hybrid_clock.cc:648] HybridClock initialized: now 1786533234856278 us; error 48 us; skew 500 ppm
Aug 12 11:13:54 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:54.859275 19345 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:13:54.860137 19345 webserver.cc:460] Webserver started at http://127.23.97.193:36687/ using document root <none> and password file <none>
I20260812 11:13:54.860754 19345 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:54.860842 19345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:54.864773 19345 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:13:54.866988 19436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:54.868300 19345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:54.868450 19345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "140966be3c0d48c6ba3cdc559fa1fab1"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "7d0b7ab15631216c0a037e6bbba7fd89"
server_key_iv: "2e4dd8a0e3304b2ba802a45b326d34b6"
server_key_version: "encryptionkey@0"
I20260812 11:13:54.869040 19345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:54.886890 19345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:54.890180 19345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:54.890398 19345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:54.890978 19345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:54.892089 19448 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:13:54.897647 19345 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:13:54.897730 19345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 11:13:54.897815 19345 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:13:54.899556 19345 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:13:54.899621 19345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 11:13:54.899930 19448 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap starting.
I20260812 11:13:54.910531 19345 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44679
I20260812 11:13:54.910910 19653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44679 every 8 connection(s)
I20260812 11:13:54.911558 19345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:13:54.919790 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 19345
I20260812 11:13:54.920288 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:36803
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=35753
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:13:54.921989 19448 log.cc:824] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:13:54.931051 19448 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:13:54.931485 19448 tablet_bootstrap.cc:492] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Bootstrap complete.
I20260812 11:13:54.933004 19448 ts_tablet_manager.cc:1397] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.010s
I20260812 11:13:54.936355 19448 raft_consensus.cc:348] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.936949 19448 raft_consensus.cc:729] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140966be3c0d48c6ba3cdc559fa1fab1, State: Initialized, Role: FOLLOWER
I20260812 11:13:54.937489 19448 consensus_queue.cc:260] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:54.938599 19448 ts_tablet_manager.cc:1428] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
W20260812 11:13:54.959239 19658 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:13:54.913543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:33535)
0812 11:13:54.914013 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.914064 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:54.914873 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:54.958341 (+ 43468us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":146,"threads_started":1}
W20260812 11:13:54.959761 19656 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:34465 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:34465: Client connection negotiation failed: client connection to 127.23.97.254:34465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:13:54.960501 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.960224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:53233)
0812 11:13:54.960350 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.960444 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 11:13:54.961133 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:54.960922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:51995)
0812 11:13:54.961030 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:54.961075 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 11:13:54.961299 19656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:13:55.036214 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:55.036509 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:55.036579 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:55.040491 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:55.040572 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:55.040660 19659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:13:55.045683 19659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:36803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=35753
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:55.046675 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:55.047663 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:55.057504 19667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:55.057662 19659 server_base.cc:1034] running on GCE node
W20260812 11:13:55.057515 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:55.057871 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:55.058250 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:55.058883 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:55.060065 19659 hybrid_clock.cc:648] HybridClock initialized: now 1786533235060049 us; error 28 us; skew 500 ppm
Aug 12 11:13:55 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533235, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:55.062693 19659 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:13:55.063529 19659 webserver.cc:460] Webserver started at http://127.23.97.194:35753/ using document root <none> and password file <none>
I20260812 11:13:55.064066 19659 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:55.064155 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:55.067823 19659 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:13:55.069926 19683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.071086 19659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:13:55.071215 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "7d344381ce274da3aa687b1589599884"
format_stamp: "Formatted at 2026-08-12 11:13:52 on dist-test-slave-zpvx"
server_key: "13f56f5bb711cc9c2bacd4d66fc8dbff"
server_key_iv: "4d0b392fbc9ad0e388dc767b57ecc313"
server_key_version: "encryptionkey@0"
I20260812 11:13:55.071738 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:55.106357 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:55.109354 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:55.109560 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:55.110165 19659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:55.111007 19659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:55.111078 19659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.111146 19659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:55.111186 19659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.120673 19659 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:36803
I20260812 11:13:55.120718 19882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:36803 every 8 connection(s)
I20260812 11:13:55.121675 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:13:55.125734 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.123792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:46211)
0812 11:13:55.124316 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.124509 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
W20260812 11:13:55.126257 19886 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:34465 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:34465: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
I20260812 11:13:55.126839 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.126625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:59501)
0812 11:13:55.126729 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.126779 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 11:13:55.127460 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.127260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:47911)
0812 11:13:55.127361 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.127405 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 11:13:55.127640 19886 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:13:55.128954 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 19659
I20260812 11:13:55.129400 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:35433
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:34465
--webserver_port=37317
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:13:55.242637 19892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:13:55.242928 19892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:13:55.243002 19892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:13:55.246946 19892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:13:55.247037 19892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:13:55.247134 19892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:13:55.252610 19892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:35433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=37317
--tserver_master_addrs=127.23.97.254:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:13:55.253719 19892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:13:55.254766 19892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:13:55.264243 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:55.264276 19902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:13:55.264243 19901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:13:55.264531 19892 server_base.cc:1034] running on GCE node
I20260812 11:13:55.265017 19892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:13:55.265726 19892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:13:55.266912 19892 hybrid_clock.cc:648] HybridClock initialized: now 1786533235266897 us; error 30 us; skew 500 ppm
Aug 12 11:13:55 dist-test-slave-zpvx krb5kdc[17261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533235, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:13:55.269721 19892 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:13:55.270557 19892 webserver.cc:460] Webserver started at http://127.23.97.195:37317/ using document root <none> and password file <none>
I20260812 11:13:55.271121 19892 fs_manager.cc:359] Metadata directory not provided
I20260812 11:13:55.271209 19892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:13:55.275353 19892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:13:55.277494 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.278689 19892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:13:55.278826 19892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "d4937befe7594f03bcd5c57988244868"
format_stamp: "Formatted at 2026-08-12 11:13:53 on dist-test-slave-zpvx"
server_key: "753ea356346f36b95ea997d9b6677799"
server_key_iv: "62cd9813af5b1a2835b0fb3805ed8233"
server_key_version: "encryptionkey@0"
I20260812 11:13:55.279392 19892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:13:55.304541 19892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:13:55.307626 19892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:13:55.307847 19892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:13:55.308533 19892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:13:55.309438 19892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:13:55.309508 19892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.309579 19892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:13:55.309619 19892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:13:55.318675 19892 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:35433
I20260812 11:13:55.318704 20125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:35433 every 8 connection(s)
I20260812 11:13:55.319626 19892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:13:55.323688 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.321866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:41191)
0812 11:13:55.322347 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.322461 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
W20260812 11:13:55.324182 20126 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:34465 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:34465: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
I20260812 11:13:55.324820 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.324545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:38755)
0812 11:13:55.324669 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.324764 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 11:13:55.325405 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.325221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:38099)
0812 11:13:55.325315 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.325359 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 11:13:55.325615 20126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:13:55.326465 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 19892
I20260812 11:13:55.962380 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:55.961883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:54147)
0812 11:13:55.962183 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:55.962251 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260812 11:13:56.128633 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.128206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:52873)
0812 11:13:56.128478 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:56.128531 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260812 11:13:56.242605 20141 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:56.242815 20141 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:56.243821 20141 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
Aug 12 11:13:56 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.195@KRBTEST.COM
Aug 12 11:13:56 dist-test-slave-zpvx krb5kdc[17261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533234, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.194@KRBTEST.COM
I20260812 11:13:56.257328 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.244044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:35433 (local address 127.23.97.193:54403)
0812 11:13:56.244158 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:56.244174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:56.244291 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:56.246033 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:56.246039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:56.246613 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:56.247072 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:56.247086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.248357 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.248360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:56.249381 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:56.249391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.250076 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.250080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:56.250353 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:56.250979 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:56.250996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:56.252615 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:56.254912 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:56.254917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:56.254931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:56.255317 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:56.255686 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:56.255694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:56.255804 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:56.256416 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:56.256422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:56.256648 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 11:13:56.256824 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 11:13:56.257138 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":194,"spinlock_wait_cycles":5760}
I20260812 11:13:56.257368 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.244197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54403 (local address 127.23.97.195:35433)
0812 11:13:56.244620 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:56.244634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:56.244673 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:13:56.245335 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:56.245340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:56.245679 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:56.245988 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:56.247242 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.248207 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.249500 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.249951 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.250122 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:56.252860 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:56.252867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:56.252884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:56.252923 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:56.254793 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255453 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:56.255461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:56.255468 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:56.255587 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255897 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:56.255902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:56.255904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:56.256278 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:56.256440 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:56.256953 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 11:13:56.257205 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":84,"threads_started":1}
I20260812 11:13:56.257400 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.244556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:32913 (local address 127.23.97.194:36803)
0812 11:13:56.244980 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 11:13:56.244999 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:13:56.245029 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:13:56.245647 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:13:56.245652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:13:56.245932 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:13:56.246199 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:13:56.247243 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.248201 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.249500 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.249952 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.250123 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:56.253145 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:13:56.253150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:13:56.253160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:13:56.253185 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:13:56.254907 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255441 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:56.255445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:56.255449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:56.255560 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255911 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:13:56.255914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:13:56.255915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:13:56.256278 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:13:56.256437 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:13:56.256948 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 11:13:56.257212 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":131,"threads_started":1}
I20260812 11:13:56.257437 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.244352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36803 (local address 127.23.97.193:32913)
0812 11:13:56.244650 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:56.244667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:13:56.244773 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:13:56.246148 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:13:56.246151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:13:56.246621 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:13:56.247072 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:56.247086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.248354 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.248357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:56.249379 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:13:56.249389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:13:56.250071 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:13:56.250075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:13:56.250351 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:13:56.251065 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:13:56.251080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:13:56.252975 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:13:56.255019 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:56.255023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:56.255314 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:56.255657 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:13:56.255661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:13:56.255663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:13:56.255799 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:13:56.256414 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:13:56.256420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:13:56.256648 (+ 228us) client_negotiation.cc:769] Sending connection context
0812 11:13:56.256824 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 11:13:56.257224 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":338,"spinlock_wait_cycles":11648,"thread_start_us":75,"threads_started":1}
W20260812 11:13:56.259979 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
W20260812 11:13:56.260699 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
I20260812 11:13:56.260795 19443 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1; no voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868
I20260812 11:13:56.261075 20141 raft_consensus.cc:2726] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:13:56.326529 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.326114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:56201)
0812 11:13:56.326377 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:56.326433 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260812 11:13:56.963554 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:56.963096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:53897)
0812 11:13:56.963393 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:56.963452 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260812 11:13:57.129846 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:57.129418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:43653)
0812 11:13:57.129684 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:57.129747 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260812 11:13:57.327595 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:57.327201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:44659)
0812 11:13:57.327437 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:57.327491 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260812 11:13:57.893009 20163 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:13:57.893200 20163 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:13:57.893651 20163 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
W20260812 11:13:57.894520 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
W20260812 11:13:57.894734 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
I20260812 11:13:57.894869 19443 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1; no voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868
I20260812 11:13:57.895071 20163 raft_consensus.cc:2726] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:13:57.964723 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:57.964298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:49453)
0812 11:13:57.964572 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:57.964627 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":112,"threads_started":1}
I20260812 11:13:58.130909 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:58.130521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:59321)
0812 11:13:58.130749 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:58.130807 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260812 11:13:58.328656 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:58.328281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:43319)
0812 11:13:58.328520 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:58.328571 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":114,"threads_started":1}
I20260812 11:13:58.965819 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:58.965386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:60041)
0812 11:13:58.965650 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:58.965701 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260812 11:13:59.132133 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:59.131607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:39051)
0812 11:13:59.131929 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:59.132012 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260812 11:13:59.329816 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:59.329418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:46617)
0812 11:13:59.329671 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:59.329726 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260812 11:13:59.967312 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:13:59.966838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:33313)
0812 11:13:59.967140 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 11:13:59.967200 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":74,"threads_started":1}
I20260812 11:14:00.133301 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:00.132899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:57715)
0812 11:14:00.133153 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:00.133208 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260812 11:14:00.331009 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:00.330554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:60597)
0812 11:14:00.330851 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:00.330913 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260812 11:14:00.968631 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:00.968153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:35237)
0812 11:14:00.968457 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:00.968518 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260812 11:14:01.036513 20181 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:01.036680 20181 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:14:01.037106 20181 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
W20260812 11:14:01.037987 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
W20260812 11:14:01.038115 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
I20260812 11:14:01.038174 19443 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1; no voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868
I20260812 11:14:01.038343 20181 raft_consensus.cc:2726] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:14:01.134402 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:01.134011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:34589)
0812 11:14:01.134250 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:01.134307 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260812 11:14:01.332361 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:01.331921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:56685)
0812 11:14:01.332199 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:01.332261 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260812 11:14:01.969843 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:01.969423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:50387)
0812 11:14:01.969685 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:01.969740 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260812 11:14:02.135629 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:02.135172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:36823)
0812 11:14:02.135473 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:02.135534 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
I20260812 11:14:02.333547 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:02.333110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:59001)
0812 11:14:02.333403 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:02.333457 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260812 11:14:02.971047 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:02.970626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:45133)
0812 11:14:02.970889 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:02.970947 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260812 11:14:03.136937 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:03.136470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:57167)
0812 11:14:03.136707 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:03.136761 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260812 11:14:03.334666 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:03.334248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:50575)
0812 11:14:03.334517 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:03.334578 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260812 11:14:03.972246 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:03.971806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:47281)
0812 11:14:03.972084 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:03.972142 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260812 11:14:04.009989 20199 raft_consensus.cc:482] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:04.010146 20199 raft_consensus.cc:504] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4937befe7594f03bcd5c57988244868" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 35433 } } peers { permanent_uuid: "7d344381ce274da3aa687b1589599884" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36803 } } peers { permanent_uuid: "140966be3c0d48c6ba3cdc559fa1fab1" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44679 } }
I20260812 11:14:04.010643 20199 leader_election.cc:288] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433), 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803)
W20260812 11:14:04.011421 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4937befe7594f03bcd5c57988244868 (127.23.97.195:35433): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
W20260812 11:14:04.011543 19443 leader_election.cc:341] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d344381ce274da3aa687b1589599884 (127.23.97.194:36803): Not found: Tablet not found: ea3412500e354d649ce2b7bcbddb3267
I20260812 11:14:04.011585 19443 leader_election.cc:302] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 140966be3c0d48c6ba3cdc559fa1fab1; no voters: 7d344381ce274da3aa687b1589599884, d4937befe7594f03bcd5c57988244868
I20260812 11:14:04.011719 20199 raft_consensus.cc:2726] T ea3412500e354d649ce2b7bcbddb3267 P 140966be3c0d48c6ba3cdc559fa1fab1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:14:04.138063 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:04.137665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:34869)
0812 11:14:04.137912 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:04.137969 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260812 11:14:04.335891 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:04.335476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:60271)
0812 11:14:04.335725 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:04.335795 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260812 11:14:04.973685 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:04.973240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:33191)
0812 11:14:04.973519 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:04.973583 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260812 11:14:05.139233 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:05.138823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:55251)
0812 11:14:05.139077 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:05.139136 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260812 11:14:05.337001 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:05.336599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:58099)
0812 11:14:05.336849 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:05.336909 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260812 11:14:05.974943 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:05.974533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:44389)
0812 11:14:05.974789 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:05.974844 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260812 11:14:06.140504 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:06.140013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:44415)
0812 11:14:06.140297 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:06.140352 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260812 11:14:06.338218 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:06.337768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:53275)
0812 11:14:06.338043 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:06.338129 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
I20260812 11:14:06.976047 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:06.975631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:40143)
0812 11:14:06.975899 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:06.975948 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260812 11:14:07.141619 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:07.141198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:51855)
0812 11:14:07.141456 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:07.141512 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260812 11:14:07.339394 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:07.338964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:39349)
0812 11:14:07.339236 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:07.339297 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260812 11:14:07.977281 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:07.976793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:34245)
0812 11:14:07.977103 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:07.977168 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260812 11:14:08.142963 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:08.142512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:54359)
0812 11:14:08.142804 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:08.142865 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260812 11:14:08.340622 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:08.340177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:60033)
0812 11:14:08.340477 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:08.340529 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260812 11:14:08.978430 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:08.978033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:54083)
0812 11:14:08.978275 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:08.978330 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260812 11:14:09.144111 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:09.143731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:58969)
0812 11:14:09.143963 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:09.144015 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260812 11:14:09.341725 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:09.341340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:47579)
0812 11:14:09.341578 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:09.341632 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260812 11:14:09.979758 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:09.979354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.193:55397)
0812 11:14:09.979600 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:09.979654 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260812 11:14:10.145526 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:10.145104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.194:55977)
0812 11:14:10.145371 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:10.145428 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260812 11:14:10.332325 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 19345
I20260812 11:14:10.343851 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:10.342766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34465 (local address 127.23.97.195:41757)
0812 11:14:10.343599 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:10.343696 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":563,"threads_started":1}
I20260812 11:14:10.351155 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 19659
I20260812 11:14:10.362318 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 19892
2026-08-12T11:14:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19593 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:14:10 dist-test-slave-zpvx krb5kdc[20242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:14:10 dist-test-slave-zpvx krb5kdc[20242](info): set up 2 sockets
Aug 12 11:14:10 dist-test-slave-zpvx krb5kdc[20242](info): commencing operation
krb5kdc: starting...
W20260812 11:14:11.683645 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:14:11 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:14:11.811228 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:11.928359 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:11.928666 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:11.928741 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:11.932713 20280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:14:11.932782 20280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:14:11.932808 20280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:14:11.932838 20280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:11.932883 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:11.932925 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:11.932962 20280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:11.932994 20280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:11.938084 20280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:11.939082 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:11.940083 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:11.948736 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:11.948747 20288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:11.948871 20280 server_base.cc:1034] running on GCE node
W20260812 11:14:11.949025 20289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:11.949697 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:11.950851 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:11.952073 20280 hybrid_clock.cc:648] HybridClock initialized: now 1786533251952050 us; error 34 us; skew 500 ppm
Aug 12 11:14:11 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533251, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:11.959540 20280 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:14:11.960512 20280 webserver.cc:460] Webserver started at http://127.23.97.254:39421/ using document root <none> and password file <none>
I20260812 11:14:11.961079 20280 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:11.961170 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:11.961416 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:11.963454 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "e5374091e05f4834b5876e21165d6dfa"
format_stamp: "Formatted at 2026-08-12 11:14:11 on dist-test-slave-zpvx"
server_key: "098d6be2f93f4dd4f9f892c07de5e4ac"
server_key_iv: "9b6029d3e69381ed9451359c81354a7f"
server_key_version: "encryptionkey@0"
I20260812 11:14:11.964093 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "e5374091e05f4834b5876e21165d6dfa"
format_stamp: "Formatted at 2026-08-12 11:14:11 on dist-test-slave-zpvx"
server_key: "098d6be2f93f4dd4f9f892c07de5e4ac"
server_key_iv: "9b6029d3e69381ed9451359c81354a7f"
server_key_version: "encryptionkey@0"
I20260812 11:14:11.968232 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:14:11.970680 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:11.971907 20280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:14:11.972059 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e5374091e05f4834b5876e21165d6dfa"
format_stamp: "Formatted at 2026-08-12 11:14:11 on dist-test-slave-zpvx"
server_key: "098d6be2f93f4dd4f9f892c07de5e4ac"
server_key_iv: "9b6029d3e69381ed9451359c81354a7f"
server_key_version: "encryptionkey@0"
I20260812 11:14:11.972168 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:11.990026 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:11.993306 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:11.993506 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:12.000511 20280 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40659
I20260812 11:14:12.000798 20406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40659 every 8 connection(s)
I20260812 11:14:12.001470 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:12.004171 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:12.007784 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 20280
I20260812 11:14:12.007949 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:14:12.008260 23943 external_mini_cluster.cc:1428] Setting key 23a741c8d31567fed3d2b8ea57cfce86
I20260812 11:14:12.010293 20407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap starting.
I20260812 11:14:12.013063 20407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:12.013949 20407 log.cc:824] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:12.015826 20407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: No bootstrap required, opened a new log
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:12.019783 20407 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:12.020033 20407 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:12.020123 20407 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Initialized, Role: FOLLOWER
I20260812 11:14:12.020603 20407 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:12.020769 20407 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:12.020854 20407 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:12.020955 20407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:12.021838 20407 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:12.022240 20407 leader_election.cc:302] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5374091e05f4834b5876e21165d6dfa; no voters:
I20260812 11:14:12.022636 20407 leader_election.cc:288] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:14:12.022733 20416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:12.022959 20416 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 LEADER]: Becoming Leader. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Running, Role: LEADER
I20260812 11:14:12.023315 20416 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:12.023945 20407 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:12.024029 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.009824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.23.97.254:40659)
0812 11:14:12.010278 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.010298 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.010344 (+ 46us) server_negotiation.cc:407] Connection header received
0812 11:14:12.011180 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.011203 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.011630 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.011993 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:12.012831 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.014365 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.015245 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.015722 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.015895 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.018489 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:12.018494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:12.018506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:12.018532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:12.020597 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.021147 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.021152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.021157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.021267 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.021559 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.021563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.021565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.021959 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:12.022163 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.022511 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.023014 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":134,"threads_started":1}
I20260812 11:14:12.026221 20420 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5374091e05f4834b5876e21165d6dfa. Latest consensus state: current_term: 1 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:12.026227 20419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:12.027274 20420 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:12.027271 20419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
W20260812 11:14:12.029410 20433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:14:12.029477 20433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:14:12.029560 20438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:12.030278 20438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:12.035698 20438 catalog_manager.cc:1347] Generated new cluster ID: 0428089388c44b5082e4a311ed17f8ee
I20260812 11:14:12.035764 20438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:12.053712 20438 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:14:12.054977 20438 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:12.061486 20438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Generated new TSK 0
I20260812 11:14:12.062214 20438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:14:12.095942 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:12.208025 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:12.208323 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:12.208410 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:12.212291 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:12.212378 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:12.212534 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:14:12.217643 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:12.218647 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:12.219666 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:12.229341 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:12.229360 20456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:12.229367 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:12.229629 20447 server_base.cc:1034] running on GCE node
I20260812 11:14:12.230110 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:12.230737 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:12.231926 20447 hybrid_clock.cc:648] HybridClock initialized: now 1786533252231916 us; error 32 us; skew 500 ppm
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:12.234730 20447 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:14:12.235565 20447 webserver.cc:460] Webserver started at http://127.23.97.193:37631/ using document root <none> and password file <none>
I20260812 11:14:12.236127 20447 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:12.236222 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:12.236485 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:12.238474 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "f441070d4a314fe0915c109e295d5cc2"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "fcbe51fa7b675610ad9febc019ec9996"
server_key_iv: "e72ec4aecf2f0ffdc71848278957ed34"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.239066 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "f441070d4a314fe0915c109e295d5cc2"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "fcbe51fa7b675610ad9febc019ec9996"
server_key_iv: "e72ec4aecf2f0ffdc71848278957ed34"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.242798 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260812 11:14:12.245235 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.246477 20447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:14:12.246616 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "f441070d4a314fe0915c109e295d5cc2"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "fcbe51fa7b675610ad9febc019ec9996"
server_key_iv: "e72ec4aecf2f0ffdc71848278957ed34"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.246733 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:12.256510 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:12.259274 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:12.259465 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:12.260025 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:12.260881 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:12.260953 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.261021 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:12.261072 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.269771 20447 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44457
I20260812 11:14:12.269796 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44457 every 8 connection(s)
I20260812 11:14:12.270576 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:14:12.272346 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 20447
I20260812 11:14:12.272528 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:14:12.272835 23943 external_mini_cluster.cc:1428] Setting key d6947bd0514d7c3a87b5c1ea33c6b3bc
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:14:12.285671 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.272307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:58133 (local address 127.23.97.254:40659)
0812 11:14:12.272456 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.272461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.273466 (+ 1005us) server_negotiation.cc:407] Connection header received
0812 11:14:12.275021 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.275026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.275087 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.275178 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:12.276420 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.277332 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.278506 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.278928 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.279057 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.281764 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:12.281770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:12.281773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:12.281807 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:12.283488 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.284024 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.284028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.284030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.284125 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.284515 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.284521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.284522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.284730 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:12.284850 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.285349 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.285489 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:14:12.286520 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.272793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:58133)
0812 11:14:12.273314 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.273372 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.274808 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.275335 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.275341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.275723 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:12.276238 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.276253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.277489 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.277492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.278370 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.278380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.279079 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.279082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.279219 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.279848 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:12.279874 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:12.281587 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:12.283620 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.283626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.283639 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.283896 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.284223 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.284226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.284227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.284358 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.284850 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:12.284858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:12.285096 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.285285 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.285635 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:14:12.287791 20681 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:12.288022 20681 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:12.288558 20681 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:12.290046 20345 ts_manager.cc:194] Registered new tserver with Master: f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457)
I20260812 11:14:12.291276 20345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:58133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:14:12.301630 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:12.412649 20687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:12.412915 20687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:12.412985 20687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:12.416986 20687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:12.417062 20687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:12.417155 20687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:14:12.422175 20687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:12.423175 20687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:12.424230 20687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:12.434923 20698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:12.434973 20687 server_base.cc:1034] running on GCE node
W20260812 11:14:12.434945 20699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:12.434906 20701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:12.435623 20687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:12.436235 20687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:12.437428 20687 hybrid_clock.cc:648] HybridClock initialized: now 1786533252437405 us; error 37 us; skew 500 ppm
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:12.440130 20687 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:14:12.440999 20687 webserver.cc:460] Webserver started at http://127.23.97.194:44383/ using document root <none> and password file <none>
I20260812 11:14:12.441501 20687 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:12.441560 20687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:12.441730 20687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:12.443634 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "c3805d29a11f0cb3bf85b0a9d8dfc7b2"
server_key_iv: "bb3281a141c373f8012840b6efc14d84"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.444154 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "c3805d29a11f0cb3bf85b0a9d8dfc7b2"
server_key_iv: "bb3281a141c373f8012840b6efc14d84"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.448004 20687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:14:12.450599 20721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.451684 20687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:12.451833 20687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "c3805d29a11f0cb3bf85b0a9d8dfc7b2"
server_key_iv: "bb3281a141c373f8012840b6efc14d84"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.451936 20687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:12.467686 20687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:12.470726 20687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:12.470925 20687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:12.471504 20687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:12.472308 20687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:12.472355 20687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.472452 20687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:12.472494 20687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.481709 20687 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:35247
I20260812 11:14:12.481734 20918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:35247 every 8 connection(s)
I20260812 11:14:12.482558 20687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:12.487567 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 20687
I20260812 11:14:12.487680 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:14:12.488001 23943 external_mini_cluster.cc:1428] Setting key e9aa77038b35269995af9a83f2f5ed98
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:12.496357 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.484321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:40701 (local address 127.23.97.254:40659)
0812 11:14:12.484512 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.484516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.485305 (+ 789us) server_negotiation.cc:407] Connection header received
0812 11:14:12.486595 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.486598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.486653 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.486762 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:12.488025 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.488731 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.489863 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.490096 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.490208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.492825 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:12.492830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:12.492833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:12.492865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:12.494262 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.494822 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.494830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.494832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.494907 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.495262 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.495266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.495268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.495457 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:12.495587 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.496062 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.496190 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:14:12.497135 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.484725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:40701)
0812 11:14:12.485151 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.485193 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.486383 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.486930 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.486937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.487288 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:12.487878 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.487894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.488843 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.488846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.489734 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.489743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.490246 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.490251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.490422 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.491005 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:12.491021 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:12.492661 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:12.494382 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.494389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.494402 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.494720 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.495001 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.495005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.495007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.495160 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.495594 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:12.495603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:12.495827 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.496016 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.496323 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260812 11:14:12.498436 20919 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:12.498678 20919 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:14:12.499178 20919 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:12.500239 20345 ts_manager.cc:194] Registered new tserver with Master: cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247)
I20260812 11:14:12.500860 20345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:40701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:14:12.517225 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:12.628077 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:12.628304 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:12.628372 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:12.632511 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:12.632562 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:12.632655 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:14:12.637676 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:12.638603 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:12.639547 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:12.648928 20943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:12.648940 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:12.648931 20950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:12.649206 20931 server_base.cc:1034] running on GCE node
I20260812 11:14:12.649649 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:12.650313 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:12.651467 20931 hybrid_clock.cc:648] HybridClock initialized: now 1786533252651457 us; error 37 us; skew 500 ppm
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:12.654184 20931 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:14:12.654994 20931 webserver.cc:460] Webserver started at http://127.23.97.195:42313/ using document root <none> and password file <none>
I20260812 11:14:12.655493 20931 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:12.655592 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:12.655829 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:14:12.657972 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "c1c67ebb8931409da653961be51eb68b"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "3b51066e072bfc09d89195605dc70c83"
server_key_iv: "c87b03535090ee59a58997e649b0fefe"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.658602 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "c1c67ebb8931409da653961be51eb68b"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "3b51066e072bfc09d89195605dc70c83"
server_key_iv: "c87b03535090ee59a58997e649b0fefe"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.662206 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:14:12.664595 20964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.665771 20931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:14:12.665891 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "c1c67ebb8931409da653961be51eb68b"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "3b51066e072bfc09d89195605dc70c83"
server_key_iv: "c87b03535090ee59a58997e649b0fefe"
server_key_version: "encryptionkey@0"
I20260812 11:14:12.665990 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:12.692164 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:12.695034 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:12.695236 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:12.695816 20931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:12.696686 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:12.696736 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.696803 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:12.696841 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:12.705912 20931 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39563
I20260812 11:14:12.705943 21169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39563 every 8 connection(s)
I20260812 11:14:12.706753 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:12.713431 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 20931
I20260812 11:14:12.713554 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:14:12.713861 23943 external_mini_cluster.cc:1428] Setting key 117b2c442d01d623f2bbbf4a77ed26a9
Aug 12 11:14:12 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:12.720077 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.708536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:37291 (local address 127.23.97.254:40659)
0812 11:14:12.708675 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.708679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.709474 (+ 795us) server_negotiation.cc:407] Connection header received
0812 11:14:12.710676 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.710679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.710729 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.710803 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:12.711886 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.712447 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.713594 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.713919 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.714035 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.716869 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:12.716872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:12.716874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:12.716897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:12.718353 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.718783 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.718786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.718787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.718860 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.719157 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.719159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.719161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.719345 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:12.719422 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.719867 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.719960 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:14:12.720898 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.708888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:37291)
0812 11:14:12.709325 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.709365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.710484 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.710961 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.710967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.711303 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:12.711757 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.711770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.712563 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.712565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.713439 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.713448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.714080 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.714083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.714230 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.714881 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:12.714897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:12.716717 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:12.718448 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.718452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.718463 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.718699 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.718944 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:12.718946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:12.718947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:12.719077 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:12.719454 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:12.719459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:12.719666 (+ 207us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.719819 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.720048 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260812 11:14:12.721997 21171 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:12.722244 21171 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:12.722700 21171 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:12.723701 20345 ts_manager.cc:194] Registered new tserver with Master: c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
I20260812 11:14:12.724253 20345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:37291
I20260812 11:14:12.727656 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:14:12.736778 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.729655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.23.97.254:40659)
0812 11:14:12.729792 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.729797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.729986 (+ 189us) server_negotiation.cc:407] Connection header received
0812 11:14:12.730088 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.730090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.730128 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.730196 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:12.730848 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.731333 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.732129 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.732336 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.732503 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.733813 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:12.733816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:12.733818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:12.733840 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:12.735112 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.735540 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.735543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.735544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.735617 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:12.735855 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:12.735858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:12.735859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:12.736057 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:12.736161 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.736461 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.736589 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:14:12.739540 20345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:12.742619 20345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:12.756616 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.751183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39563 (local address 127.0.0.1:46898)
0812 11:14:12.751758 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.751775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.751866 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.752577 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.752581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.752603 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:12.752699 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.752705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.753931 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.753934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.754951 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.754961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.755860 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.755865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.755993 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.756035 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.756154 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.756271 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260812 11:14:12.756649 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.751632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.23.97.195:39563)
0812 11:14:12.752098 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.752120 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.752143 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:14:12.752228 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.752233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.752434 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.752579 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:12.752833 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.753802 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.755096 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.755667 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.755800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.755945 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.756362 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.756501 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":161,"threads_started":1}
I20260812 11:14:12.757056 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.750528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.23.97.194:35247)
0812 11:14:12.751011 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.751025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.751053 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:14:12.751136 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.751142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.751366 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.751530 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:12.752083 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.753319 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.755091 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.756038 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.756138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.756269 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.756810 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.756908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":118,"threads_started":1}
I20260812 11:14:12.756692 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.750430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35247 (local address 127.0.0.1:57872)
0812 11:14:12.750787 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.750838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.750973 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.751753 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.751763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.751794 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:12.751921 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.751929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.753462 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.753465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.754954 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.754964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.756158 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.756161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.756241 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.756262 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.756343 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.756420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260812 11:14:12.757476 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.751261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44457 (local address 127.0.0.1:57560)
0812 11:14:12.752032 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.752044 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.752116 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.752969 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.752972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.752989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:12.753085 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.753092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.754479 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.754482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.755813 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.755825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.757107 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.757111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.757195 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.757229 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.757300 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.757361 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":94,"threads_started":1}
I20260812 11:14:12.757932 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.751672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.23.97.193:44457)
0812 11:14:12.752528 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.752547 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.752571 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:14:12.752643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.752648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.752827 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.752981 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:12.753178 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.754307 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.755961 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.756804 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.756904 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.757038 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.757702 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.757793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":97,"threads_started":1}
I20260812 11:14:12.759806 21041 tablet_service.cc:1467] Processing CreateTablet for tablet e05c62ef078346ea9e64b9351c43b49d (DEFAULT_TABLE table=test-table [id=b7e2443a9a0f40979019f4912e98f86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:12.759974 20812 tablet_service.cc:1467] Processing CreateTablet for tablet e05c62ef078346ea9e64b9351c43b49d (DEFAULT_TABLE table=test-table [id=b7e2443a9a0f40979019f4912e98f86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:12.760902 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05c62ef078346ea9e64b9351c43b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:12.760571 20561 tablet_service.cc:1467] Processing CreateTablet for tablet e05c62ef078346ea9e64b9351c43b49d (DEFAULT_TABLE table=test-table [id=b7e2443a9a0f40979019f4912e98f86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:12.761075 20812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05c62ef078346ea9e64b9351c43b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:12.761574 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05c62ef078346ea9e64b9351c43b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:12.767422 21200 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Bootstrap starting.
I20260812 11:14:12.768472 21201 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Bootstrap starting.
I20260812 11:14:12.768875 21202 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Bootstrap starting.
I20260812 11:14:12.770332 21200 tablet_bootstrap.cc:654] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:12.771108 21202 tablet_bootstrap.cc:654] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:12.771226 21200 log.cc:824] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:12.772337 21202 log.cc:824] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:12.773449 21200 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: No bootstrap required, opened a new log
I20260812 11:14:12.773667 21200 ts_tablet_manager.cc:1397] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 11:14:12.774119 21201 tablet_bootstrap.cc:654] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:12.774497 21202 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: No bootstrap required, opened a new log
I20260812 11:14:12.774679 21202 ts_tablet_manager.cc:1397] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:14:12.774996 21201 log.cc:824] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:12.777127 21201 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: No bootstrap required, opened a new log
I20260812 11:14:12.777083 21200 raft_consensus.cc:348] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.777330 21201 ts_tablet_manager.cc:1397] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260812 11:14:12.777374 21200 raft_consensus.cc:374] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:12.777451 21200 raft_consensus.cc:729] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c67ebb8931409da653961be51eb68b, State: Initialized, Role: FOLLOWER
I20260812 11:14:12.777904 21200 consensus_queue.cc:260] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.778587 21171 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
I20260812 11:14:12.779071 21200 ts_tablet_manager.cc:1428] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 11:14:12.779762 21202 raft_consensus.cc:348] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.780015 21202 raft_consensus.cc:374] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:12.780109 21202 raft_consensus.cc:729] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f441070d4a314fe0915c109e295d5cc2, State: Initialized, Role: FOLLOWER
I20260812 11:14:12.780570 21202 consensus_queue.cc:260] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.781265 20681 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
I20260812 11:14:12.781100 21201 raft_consensus.cc:348] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.781358 21201 raft_consensus.cc:374] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:12.781417 21201 raft_consensus.cc:729] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf950bd8bf6044c1ac91a16fc44e34c9, State: Initialized, Role: FOLLOWER
I20260812 11:14:12.781553 21202 ts_tablet_manager.cc:1428] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:14:12.781816 21201 consensus_queue.cc:260] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.782604 20919 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
I20260812 11:14:12.782918 21201 ts_tablet_manager.cc:1428] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260812 11:14:12.801767 21210 raft_consensus.cc:482] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:12.801960 21210 raft_consensus.cc:504] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.802920 21210 leader_election.cc:288] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
I20260812 11:14:12.807839 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.803640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:37111 (local address 127.23.97.195:39563)
0812 11:14:12.803754 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.803759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.803985 (+ 226us) server_negotiation.cc:407] Connection header received
0812 11:14:12.804083 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.804085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.804143 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.804242 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:12.804501 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.805204 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.806600 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.807180 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.807292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.807331 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.807647 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.807718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 11:14:12.807840 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.803253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54413 (local address 127.23.97.194:35247)
0812 11:14:12.803404 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.803409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.803507 (+ 98us) server_negotiation.cc:407] Connection header received
0812 11:14:12.803683 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.803688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.803749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.803841 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:12.804156 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.804993 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.806517 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.807149 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.807280 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.807331 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.807647 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.807726 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:14:12.807893 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.803540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39563 (local address 127.23.97.193:37111)
0812 11:14:12.803887 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.803899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.803991 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.804249 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.804251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.804266 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:12.804353 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.804357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.805302 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.805305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.806475 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.806487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.807289 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.807292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.807473 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.807541 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.807611 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.807728 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":108,"thread_start_us":122,"threads_started":1}
I20260812 11:14:12.807898 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.803157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35247 (local address 127.23.97.193:54413)
0812 11:14:12.803409 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.803425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:12.803554 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:12.803871 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:12.803875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:12.803896 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:12.804049 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.804057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.805093 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.805095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.806392 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:12.806404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.807288 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.807291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:12.807478 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.807528 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:14:12.807611 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:14:12.807718 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":112,"thread_start_us":93,"threads_started":1}
I20260812 11:14:12.808538 21077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05c62ef078346ea9e64b9351c43b49d" candidate_uuid: "f441070d4a314fe0915c109e295d5cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c67ebb8931409da653961be51eb68b" is_pre_election: true
I20260812 11:14:12.808851 21077 raft_consensus.cc:2454] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f441070d4a314fe0915c109e295d5cc2 in term 0.
I20260812 11:14:12.808884 20849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05c62ef078346ea9e64b9351c43b49d" candidate_uuid: "f441070d4a314fe0915c109e295d5cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" is_pre_election: true
I20260812 11:14:12.809136 20849 raft_consensus.cc:2454] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f441070d4a314fe0915c109e295d5cc2 in term 0.
I20260812 11:14:12.809381 20477 leader_election.cc:302] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1c67ebb8931409da653961be51eb68b, f441070d4a314fe0915c109e295d5cc2; no voters:
I20260812 11:14:12.809641 21210 raft_consensus.cc:2781] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:12.809758 21210 raft_consensus.cc:482] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:12.809819 21210 raft_consensus.cc:3037] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:12.811028 21210 raft_consensus.cc:504] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.811304 21210 leader_election.cc:288] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 1 election: Requested vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
I20260812 11:14:12.811669 20849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05c62ef078346ea9e64b9351c43b49d" candidate_uuid: "f441070d4a314fe0915c109e295d5cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
I20260812 11:14:12.811776 21077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05c62ef078346ea9e64b9351c43b49d" candidate_uuid: "f441070d4a314fe0915c109e295d5cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c67ebb8931409da653961be51eb68b"
I20260812 11:14:12.811821 20849 raft_consensus.cc:3037] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:12.811929 21077 raft_consensus.cc:3037] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:12.812937 20849 raft_consensus.cc:2454] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f441070d4a314fe0915c109e295d5cc2 in term 1.
I20260812 11:14:12.813030 21077 raft_consensus.cc:2454] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f441070d4a314fe0915c109e295d5cc2 in term 1.
I20260812 11:14:12.813270 20477 leader_election.cc:302] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf950bd8bf6044c1ac91a16fc44e34c9, f441070d4a314fe0915c109e295d5cc2; no voters:
I20260812 11:14:12.813455 21210 raft_consensus.cc:2781] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:12.813678 21210 raft_consensus.cc:686] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 LEADER]: Becoming Leader. State: Replica: f441070d4a314fe0915c109e295d5cc2, State: Running, Role: LEADER
I20260812 11:14:12.814019 21210 consensus_queue.cc:237] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:12.817155 20345 catalog_manager.cc:5495] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f441070d4a314fe0915c109e295d5cc2 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "f441070d4a314fe0915c109e295d5cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } health_report { overall_health: HEALTHY } } }
I20260812 11:14:12.833771 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.830474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.23.97.193:44457)
0812 11:14:12.830590 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:12.830593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:12.830606 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:14:12.830760 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:12.830762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:12.830815 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:12.830901 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:12.831119 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.831597 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.832476 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:12.832677 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:12.832761 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:12.833049 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:12.833424 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:12.833510 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:12.833614 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 11:14:12.833662 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:14:12.839572 20849 raft_consensus.cc:1261] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Refusing update from remote peer f441070d4a314fe0915c109e295d5cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:12.839572 21077 raft_consensus.cc:1261] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Refusing update from remote peer f441070d4a314fe0915c109e295d5cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:12.840310 21221 consensus_queue.cc:1035] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:12.840555 21210 consensus_queue.cc:1035] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:12.848071 21237 mvcc.cc:205] Tried to move back new op lower bound from 7317640203623837696 to 7317640203529129984. Current Snapshot: MvccSnapshot[applied={T|T < 7317640203623837696}]
I20260812 11:14:12.849141 21235 mvcc.cc:205] Tried to move back new op lower bound from 7317640203623837696 to 7317640203529129984. Current Snapshot: MvccSnapshot[applied={T|T < 7317640203623837696}]
I20260812 11:14:12.856554 20343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:12.856730 20343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:12.858978 20343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Sending DeleteTablet for 3 replicas of tablet e05c62ef078346ea9e64b9351c43b49d
I20260812 11:14:12.859804 20812 tablet_service.cc:1514] Processing DeleteTablet for tablet e05c62ef078346ea9e64b9351c43b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:57872
I20260812 11:14:12.859858 21041 tablet_service.cc:1514] Processing DeleteTablet for tablet e05c62ef078346ea9e64b9351c43b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:46898
I20260812 11:14:12.859864 20561 tablet_service.cc:1514] Processing DeleteTablet for tablet e05c62ef078346ea9e64b9351c43b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:57560
I20260812 11:14:12.860266 21256 tablet_replica.cc:330] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: stopping tablet replica
I20260812 11:14:12.860468 21258 tablet_replica.cc:330] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: stopping tablet replica
I20260812 11:14:12.860474 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 20447
I20260812 11:14:12.877589 21256 raft_consensus.cc:2229] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:12.878178 21256 raft_consensus.cc:2258] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:12.880290 21256 ts_tablet_manager.cc:1905] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:12.882570 21256 ts_tablet_manager.cc:1918] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:12.882699 21256 log.cc:1196] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/e05c62ef078346ea9e64b9351c43b49d
I20260812 11:14:12.882926 21255 tablet_replica.cc:330] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: stopping tablet replica
I20260812 11:14:12.883102 21256 ts_tablet_manager.cc:1939] T e05c62ef078346ea9e64b9351c43b49d P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting consensus metadata
I20260812 11:14:12.883320 21255 raft_consensus.cc:2229] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:12.883659 21255 raft_consensus.cc:2258] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:12.884527 20312 catalog_manager.cc:4841] TS cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): tablet e05c62ef078346ea9e64b9351c43b49d (table test-table [id=b7e2443a9a0f40979019f4912e98f86e]) successfully deleted
I20260812 11:14:12.885210 21255 ts_tablet_manager.cc:1905] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 11:14:12.887167 20306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44457 (error 108)
I20260812 11:14:12.887378 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 20687
I20260812 11:14:12.887558 21255 ts_tablet_manager.cc:1918] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:12.887650 21255 log.cc:1196] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/e05c62ef078346ea9e64b9351c43b49d
I20260812 11:14:12.887948 21255 ts_tablet_manager.cc:1939] T e05c62ef078346ea9e64b9351c43b49d P c1c67ebb8931409da653961be51eb68b: Deleting consensus metadata
I20260812 11:14:12.889709 20312 catalog_manager.cc:4841] TS c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): tablet e05c62ef078346ea9e64b9351c43b49d (table test-table [id=b7e2443a9a0f40979019f4912e98f86e]) successfully deleted
I20260812 11:14:12.897945 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:12.897570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44457 (local address 127.0.0.1:57580)
0812 11:14:12.897743 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:12.897833 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:14:12.898149 20306 catalog_manager.cc:4584] TS f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e05c62ef078346ea9e64b9351c43b49d: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44457: connect: Connection refused (error 111)
I20260812 11:14:12.902974 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 20931
I20260812 11:14:12.915846 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 20280
I20260812 11:14:32.932089 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--webserver_interface=127.23.97.254
--webserver_port=39421
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:33.064183 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:33.064628 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:33.064721 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:33.068962 21261 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:14:33.069043 21261 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:14:33.069084 21261 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:14:33.069105 21261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:33.069191 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:33.069243 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:33.069288 21261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:33.069329 21261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:33.074909 21261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:33.076262 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:33.077387 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:33.093046 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.092991 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.092991 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:33.093191 21261 server_base.cc:1034] running on GCE node
I20260812 11:14:33.094242 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:33.095999 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:33.097258 21261 hybrid_clock.cc:648] HybridClock initialized: now 1786533273097241 us; error 62 us; skew 500 ppm
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:33.105604 21261 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:14:33.106678 21261 webserver.cc:460] Webserver started at http://127.23.97.254:39421/ using document root <none> and password file <none>
I20260812 11:14:33.107299 21261 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:33.107403 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:33.112033 21261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:33.114673 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.116303 21261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:14:33.116485 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e5374091e05f4834b5876e21165d6dfa"
format_stamp: "Formatted at 2026-08-12 11:14:11 on dist-test-slave-zpvx"
server_key: "098d6be2f93f4dd4f9f892c07de5e4ac"
server_key_iv: "9b6029d3e69381ed9451359c81354a7f"
server_key_version: "encryptionkey@0"
I20260812 11:14:33.117091 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:33.133972 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:33.137367 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:33.137578 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:33.146338 21261 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40659
I20260812 11:14:33.146658 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40659 every 8 connection(s)
I20260812 11:14:33.147624 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:33.151891 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 21261
I20260812 11:14:33.152477 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:44457
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=37631
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:14:33.152763 21382 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:14:33.154673 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap starting.
I20260812 11:14:33.180684 21382 log.cc:824] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:33.192826 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:14:33.193313 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap complete.
I20260812 11:14:33.198590 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:33.199179 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Initialized, Role: FOLLOWER
I20260812 11:14:33.199694 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:33.199844 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:33.199920 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:33.200030 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:14:33.202545 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:33.202984 21382 leader_election.cc:302] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5374091e05f4834b5876e21165d6dfa; no voters:
I20260812 11:14:33.203344 21382 leader_election.cc:288] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:14:33.203478 21388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:14:33.203747 21388 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 LEADER]: Becoming Leader. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Running, Role: LEADER
I20260812 11:14:33.204125 21388 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:33.204811 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:33.205322 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5374091e05f4834b5876e21165d6dfa. Latest consensus state: current_term: 2 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:33.205556 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:33.205977 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:33.205986 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:33.206149 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:33.211298 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:33.211892 21400 catalog_manager.cc:1259] Loaded cluster ID: 0428089388c44b5082e4a311ed17f8ee
I20260812 11:14:33.211962 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:33.214533 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:33.215236 21400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Loaded TSK: 0
I20260812 11:14:33.234268 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Generated new TSK 1
I20260812 11:14:33.234583 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:14:33.296264 21385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:33.296579 21385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:33.296662 21385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:33.300873 21385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:33.300927 21385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:33.301009 21385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:14:33.306617 21385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:44457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=37631
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:33.307991 21385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:33.309104 21385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:33.319605 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.319628 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:33.319775 21385 server_base.cc:1034] running on GCE node
W20260812 11:14:33.320004 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:33.320442 21385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:33.321050 21385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:33.322245 21385 hybrid_clock.cc:648] HybridClock initialized: now 1786533273322193 us; error 71 us; skew 500 ppm
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:33.325762 21385 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:14:33.327303 21385 webserver.cc:460] Webserver started at http://127.23.97.193:37631/ using document root <none> and password file <none>
I20260812 11:14:33.328392 21385 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:33.328567 21385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:33.334843 21385 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:14:33.337596 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.339205 21385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:14:33.339336 21385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "f441070d4a314fe0915c109e295d5cc2"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "fcbe51fa7b675610ad9febc019ec9996"
server_key_iv: "e72ec4aecf2f0ffdc71848278957ed34"
server_key_version: "encryptionkey@0"
I20260812 11:14:33.340062 21385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:33.356972 21385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:33.361151 21385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:33.361421 21385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:33.362354 21385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:33.364265 21467 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:14:33.368896 21385 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:14:33.368973 21385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 11:14:33.369042 21385 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:14:33.371692 21385 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:14:33.371757 21385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260812 11:14:33.372129 21467 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Bootstrap starting.
I20260812 11:14:33.385543 21385 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44457
I20260812 11:14:33.385635 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44457 every 8 connection(s)
I20260812 11:14:33.386713 21385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:14:33.391892 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 21385
I20260812 11:14:33.392493 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:35247
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=44383
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:14:33.392819 21467 log.cc:824] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:33.402361 21467 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:14:33.402887 21467 tablet_bootstrap.cc:492] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Bootstrap complete.
I20260812 11:14:33.404536 21467 ts_tablet_manager.cc:1397] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.013s
I20260812 11:14:33.408711 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.390135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:51725 (local address 127.23.97.254:40659)
0812 11:14:33.391228 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.391244 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.391680 (+ 436us) server_negotiation.cc:407] Connection header received
0812 11:14:33.393183 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.393205 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.393672 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.394003 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.396521 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.398007 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.399334 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.399867 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.400197 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.402976 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.402984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.403002 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.403033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.404993 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.405534 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.405540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.405544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.405660 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.406275 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.406280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.406281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.406664 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.406813 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.407426 (+ 613us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.407750 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":707,"threads_started":1}
I20260812 11:14:33.408711 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.389503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:51725)
0812 11:14:33.391498 (+ 1995us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.391543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.392665 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.394851 (+ 2186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.394863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.395436 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:33.396275 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.396297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.398233 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.398236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.399134 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.399145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.400138 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.400143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.400454 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.401151 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:33.401168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:33.402767 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:33.405112 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.405117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.405130 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.405398 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.405849 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.405855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.405894 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.406161 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.406785 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:33.406791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:33.407057 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.407239 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.407750 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":1251,"threads_started":1}
I20260812 11:14:33.408835 21467 raft_consensus.cc:348] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.409534 21467 raft_consensus.cc:729] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f441070d4a314fe0915c109e295d5cc2, State: Initialized, Role: FOLLOWER
I20260812 11:14:33.410310 21467 consensus_queue.cc:260] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.411491 21467 ts_tablet_manager.cc:1428] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 11:14:33.411983 21678 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:33.412237 21678 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:33.412817 21678 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:33.415791 21316 ts_manager.cc:194] Registered new tserver with Master: f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457)
I20260812 11:14:33.420904 21316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:51725
I20260812 11:14:33.424350 21678 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.193@KRBTEST.COM
I20260812 11:14:33.432155 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.419663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44457 (local address 127.0.0.1:42586)
0812 11:14:33.420071 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.420113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.420238 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.421314 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.421322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.421858 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:33.421980 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.421987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.423230 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.423234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.424347 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.424360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.425228 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.425232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.425339 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.426142 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:33.426155 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:33.427581 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:33.430276 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.430284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.430289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.430507 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.430870 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.430873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.430874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.430967 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.431598 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:33.431607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:33.431747 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.431886 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.432029 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
I20260812 11:14:33.432354 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.419730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.23.97.193:44457)
0812 11:14:33.420618 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.420629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.420678 (+ 49us) server_negotiation.cc:407] Connection header received
0812 11:14:33.420755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.420759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.420909 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.421114 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.422104 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.423066 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.424559 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.425075 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.425239 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.427731 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.427740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.427753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.427779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.430159 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.430623 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.430629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.430630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.430738 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.431074 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.431077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.431079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.431458 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.431589 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.431995 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.432172 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":135,"threads_started":1}
I20260812 11:14:33.433710 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet e05c62ef078346ea9e64b9351c43b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:12 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:42586
I20260812 11:14:33.434549 21690 tablet_replica.cc:330] stopping tablet replica
I20260812 11:14:33.434845 21690 raft_consensus.cc:2229] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:33.435092 21690 raft_consensus.cc:2258] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:33.436115 21690 ts_tablet_manager.cc:1905] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:33.439817 21690 ts_tablet_manager.cc:1918] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:33.439911 21690 log.cc:1196] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal/wals/e05c62ef078346ea9e64b9351c43b49d
I20260812 11:14:33.440210 21690 ts_tablet_manager.cc:1939] T e05c62ef078346ea9e64b9351c43b49d P f441070d4a314fe0915c109e295d5cc2: Deleting consensus metadata
I20260812 11:14:33.442036 21287 catalog_manager.cc:4841] TS f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457): tablet e05c62ef078346ea9e64b9351c43b49d (table test-table [id=b7e2443a9a0f40979019f4912e98f86e]) successfully deleted
W20260812 11:14:33.526935 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:33.527204 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:33.527247 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:33.531329 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:33.531400 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:33.531456 21684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:14:33.536525 21684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:35247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=44383
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:33.537472 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:33.538429 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:33.548799 21698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.548816 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.548815 21699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:33.549325 21684 server_base.cc:1034] running on GCE node
I20260812 11:14:33.549787 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:33.550462 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:33.551651 21684 hybrid_clock.cc:648] HybridClock initialized: now 1786533273551634 us; error 41 us; skew 500 ppm
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:33.554486 21684 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:14:33.555475 21684 webserver.cc:460] Webserver started at http://127.23.97.194:44383/ using document root <none> and password file <none>
I20260812 11:14:33.556054 21684 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:33.556169 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:33.560138 21684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:14:33.562387 21718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.563752 21684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:33.563900 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "c3805d29a11f0cb3bf85b0a9d8dfc7b2"
server_key_iv: "bb3281a141c373f8012840b6efc14d84"
server_key_version: "encryptionkey@0"
I20260812 11:14:33.564525 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:33.575904 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:33.578881 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:33.579082 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:33.579650 21684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:33.580507 21684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:33.580583 21684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.580655 21684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:33.580698 21684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.590134 21684 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:35247
I20260812 11:14:33.590166 21914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:35247 every 8 connection(s)
I20260812 11:14:33.591120 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:33.594388 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 21684
I20260812 11:14:33.594889 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:39563
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=42313
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:33.607100 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.593034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:32983 (local address 127.23.97.254:40659)
0812 11:14:33.593215 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.593218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.594004 (+ 786us) server_negotiation.cc:407] Connection header received
0812 11:14:33.595654 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.595658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.595722 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.595832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.597437 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.597968 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.599294 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.599763 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.599900 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.602838 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.602842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.602844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.602869 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.604485 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.605185 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.605188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.605189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.605327 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.605815 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.605818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.605819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.605996 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.606078 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.606744 (+ 666us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.606920 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:14:33.608263 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.593412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:32983)
0812 11:14:33.593854 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.593896 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.595379 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.596007 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.596013 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.596358 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:33.597201 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.597239 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.598106 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.598110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.599151 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.599161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.599947 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.599951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.600085 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.600724 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:33.600740 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:33.602637 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:33.604628 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.604638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.604657 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.605066 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.605432 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.605439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.605442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.605640 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.606109 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:33.606117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:33.606473 (+ 356us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.606708 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.607119 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260812 11:14:33.609557 21917 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:33.609805 21917 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:33.610369 21917 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:33.611568 21316 ts_manager.cc:194] Registered new tserver with Master: cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247)
I20260812 11:14:33.612185 21316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:32983
W20260812 11:14:33.715865 21922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:33.716092 21922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:33.716161 21922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:33.720247 21922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:33.720302 21922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:33.720425 21922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:14:33.726027 21922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:39563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=42313
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:33.726954 21922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:33.728016 21922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:33.737926 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.737895 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:33.737910 21930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:33.738101 21922 server_base.cc:1034] running on GCE node
I20260812 11:14:33.738481 21922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:33.739130 21922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:33.740345 21922 hybrid_clock.cc:648] HybridClock initialized: now 1786533273740307 us; error 48 us; skew 500 ppm
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:33.743798 21922 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:14:33.744954 21922 webserver.cc:460] Webserver started at http://127.23.97.195:42313/ using document root <none> and password file <none>
I20260812 11:14:33.745612 21922 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:33.745725 21922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:33.749619 21922 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:14:33.751727 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.753181 21922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:14:33.753319 21922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "c1c67ebb8931409da653961be51eb68b"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "3b51066e072bfc09d89195605dc70c83"
server_key_iv: "c87b03535090ee59a58997e649b0fefe"
server_key_version: "encryptionkey@0"
I20260812 11:14:33.753851 21922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:33.775542 21922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:33.778704 21922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:33.778909 21922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:33.779515 21922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:33.780393 21922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:33.780467 21922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.780540 21922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:33.780592 21922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:33.790318 21922 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39563
I20260812 11:14:33.790359 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39563 every 8 connection(s)
I20260812 11:14:33.791384 21922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:33.792945 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 21922
Aug 12 11:14:33 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:14:33.803719 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.793704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.23.97.254:40659)
0812 11:14:33.794131 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.794139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.794159 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:14:33.794229 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.794233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.794288 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.794390 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.795503 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.796089 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.797453 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.797699 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.797783 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.799339 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.799343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.799346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.799379 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.801110 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.801608 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.801611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.801612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.801700 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.802495 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.802498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.802500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.802825 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.803063 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.803283 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.803419 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":1280,"thread_start_us":117,"threads_started":1}
I20260812 11:14:33.806792 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.793432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:51413 (local address 127.23.97.254:40659)
0812 11:14:33.793564 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.793568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.794686 (+ 1118us) server_negotiation.cc:407] Connection header received
0812 11:14:33.796248 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.796253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.796306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.796468 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.797913 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.798371 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.799580 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.799792 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.799899 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.802491 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.802495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.802497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.802536 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.804306 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.805033 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.805036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.805038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.805121 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.805497 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.805500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.805501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.805701 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.805826 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.806474 (+ 648us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.806620 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":4480}
I20260812 11:14:33.807821 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.793943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:51413)
0812 11:14:33.794509 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.794571 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.796040 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.796673 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.796681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.797148 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:33.797736 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.797780 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.798497 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.798501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.799432 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.799442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.799939 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.799943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.800124 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.800674 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:33.800690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:33.802313 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:33.804507 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.804511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.804523 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.804925 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.805214 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:33.805218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:33.805220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:33.805401 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:33.805905 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:33.805911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:33.806221 (+ 310us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.806434 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.806768 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260812 11:14:33.808912 22152 heartbeater.cc:347] Connected to a master server at 127.23.97.254:40659
I20260812 11:14:33.809170 22152 heartbeater.cc:464] Registering TS with master...
I20260812 11:14:33.809629 22152 heartbeater.cc:511] Master 127.23.97.254:40659 requested a full tablet report, sending...
I20260812 11:14:33.810582 21316 ts_manager.cc:194] Registered new tserver with Master: c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
I20260812 11:14:33.811137 21316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:51413
I20260812 11:14:33.817579 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:14:33.821584 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.818347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.23.97.254:40659)
0812 11:14:33.818467 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.818470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.818629 (+ 159us) server_negotiation.cc:407] Connection header received
0812 11:14:33.818752 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.818754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.818844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.818927 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:33.819107 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.819595 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.820518 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.820798 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.820878 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.821151 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:33.821279 (+ 128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 11:14:33.821467 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56650: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 11:14:33.821646 21682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56650: token expired
W20260812 11:14:33.822018 21178 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.97.254:40659)
I20260812 11:14:33.829938 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.822463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.23.97.254:40659)
0812 11:14:33.822584 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.822590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.822606 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:14:33.822755 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.822758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.822801 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.822881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:33.823491 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.823960 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.824881 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.825120 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.825226 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.826630 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:33.826633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:33.826636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:33.826661 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:33.828086 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.828571 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.828575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.828577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.828664 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:33.828983 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:33.828986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:33.828988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:33.829189 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:33.829339 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.829577 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.829708 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:14:33.832338 21316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:14:33.835780 21316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:14:33.843446 21544 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc1363dbba44b6b97601406e4bd801 (DEFAULT_TABLE table=test-table [id=d2cb1a09ef50455eb03414cc68d3dc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:33.844062 21544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc1363dbba44b6b97601406e4bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:33.847415 21467 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Bootstrap starting.
I20260812 11:14:33.847535 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.842322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35247 (local address 127.0.0.1:34948)
0812 11:14:33.842477 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.842492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.842571 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.843368 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.843371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.843390 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:33.843495 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.843502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.844978 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.844982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.845873 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.845883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.847026 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.847029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.847137 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.847186 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.847316 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.847407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 11:14:33.847766 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.842493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.23.97.194:35247)
0812 11:14:33.842833 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.842850 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.842877 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:14:33.842965 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.842971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.843203 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.843383 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:33.843617 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.844858 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.845991 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.846891 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.847025 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.847170 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.847458 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.847555 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":122,"threads_started":1}
I20260812 11:14:33.848417 21467 tablet_bootstrap.cc:654] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:33.849344 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.842799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39563 (local address 127.0.0.1:47706)
0812 11:14:33.843036 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.843052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.843137 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.844528 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.844531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.844559 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:33.844664 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.844670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.846199 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.846203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.847575 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.847585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.848946 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.848950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.849037 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.849058 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.849157 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.849226 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260812 11:14:33.849681 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.843237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.23.97.195:39563)
0812 11:14:33.843780 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.843793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.843818 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:14:33.843891 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.843896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.844079 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.844236 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:33.844783 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.846031 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.847765 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.848599 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.848744 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.848880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.849219 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.849326 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":174,"threads_started":1}
I20260812 11:14:33.850366 21467 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: No bootstrap required, opened a new log
I20260812 11:14:33.850456 21467 ts_tablet_manager.cc:1397] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 11:14:33.850840 21467 raft_consensus.cc:348] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.850925 21467 raft_consensus.cc:374] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:33.850952 21467 raft_consensus.cc:729] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f441070d4a314fe0915c109e295d5cc2, State: Initialized, Role: FOLLOWER
I20260812 11:14:33.851084 21467 consensus_queue.cc:260] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.851356 21467 ts_tablet_manager.cc:1428] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 11:14:33.850948 21798 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc1363dbba44b6b97601406e4bd801 (DEFAULT_TABLE table=test-table [id=d2cb1a09ef50455eb03414cc68d3dc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:33.852435 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc1363dbba44b6b97601406e4bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:33.852849 22023 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc1363dbba44b6b97601406e4bd801 (DEFAULT_TABLE table=test-table [id=d2cb1a09ef50455eb03414cc68d3dc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:14:33.853861 22023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc1363dbba44b6b97601406e4bd801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:14:33.861105 22178 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Bootstrap starting.
I20260812 11:14:33.861965 22182 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Bootstrap starting.
I20260812 11:14:33.863940 22178 tablet_bootstrap.cc:654] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:33.864657 22182 tablet_bootstrap.cc:654] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Neither blocks nor log segments found. Creating new log.
I20260812 11:14:33.865021 22178 log.cc:824] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:33.865664 22182 log.cc:824] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:33.867463 22178 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: No bootstrap required, opened a new log
I20260812 11:14:33.867676 22178 ts_tablet_manager.cc:1397] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:14:33.867870 22182 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: No bootstrap required, opened a new log
I20260812 11:14:33.868104 22182 ts_tablet_manager.cc:1397] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260812 11:14:33.872043 22178 raft_consensus.cc:348] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.872120 22182 raft_consensus.cc:348] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.872404 22178 raft_consensus.cc:374] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:33.872404 22182 raft_consensus.cc:374] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:14:33.872462 22178 raft_consensus.cc:729] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf950bd8bf6044c1ac91a16fc44e34c9, State: Initialized, Role: FOLLOWER
I20260812 11:14:33.872462 22182 raft_consensus.cc:729] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c67ebb8931409da653961be51eb68b, State: Initialized, Role: FOLLOWER
I20260812 11:14:33.872937 22178 consensus_queue.cc:260] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.872937 22182 consensus_queue.cc:260] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.873637 21917 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
I20260812 11:14:33.873878 22152 heartbeater.cc:503] Master 127.23.97.254:40659 was elected leader, sending a full tablet report...
I20260812 11:14:33.873975 22178 ts_tablet_manager.cc:1428] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 11:14:33.873975 22182 ts_tablet_manager.cc:1428] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260812 11:14:33.888082 21679 tablet.cc:2369] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:14:33.921232 22185 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:33.921428 22185 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.922590 22185 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457)
I20260812 11:14:33.926574 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.922826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35247 (local address 127.23.97.195:54525)
0812 11:14:33.923056 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.923071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.923182 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.923464 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.923467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.923484 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:33.923602 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.923607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.924565 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.924569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.925434 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.925445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.926099 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.926103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.926212 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.926231 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.926326 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.926435 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 11:14:33.926575 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.922951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:54525 (local address 127.23.97.194:35247)
0812 11:14:33.923073 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.923078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.923260 (+ 182us) server_negotiation.cc:407] Connection header received
0812 11:14:33.923313 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.923315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.923359 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.923449 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:33.923721 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.924282 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.925566 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.925974 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.926106 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.926161 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.926344 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.926452 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:14:33.926832 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.922934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:57449 (local address 127.23.97.193:44457)
0812 11:14:33.923082 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.923087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.923383 (+ 296us) server_negotiation.cc:407] Connection header received
0812 11:14:33.923547 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.923550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.923608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.923713 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:14:33.923991 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.924706 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.925807 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.926239 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.926388 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.926577 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.926621 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.926709 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:14:33.927126 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.922826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44457 (local address 127.23.97.195:57449)
0812 11:14:33.923269 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:33.923281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:33.923416 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:33.923726 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:33.923729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:33.923746 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:14:33.923872 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.923878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.924822 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.924827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.925672 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:33.925682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.926367 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.926383 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:33.926501 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.926523 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:14:33.926602 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:14:33.926968 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":21,"thread_start_us":152,"threads_started":1}
I20260812 11:14:33.927469 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc1363dbba44b6b97601406e4bd801" candidate_uuid: "c1c67ebb8931409da653961be51eb68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" is_pre_election: true
I20260812 11:14:33.927913 21840 raft_consensus.cc:2454] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c67ebb8931409da653961be51eb68b in term 0.
I20260812 11:14:33.927924 21583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc1363dbba44b6b97601406e4bd801" candidate_uuid: "c1c67ebb8931409da653961be51eb68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f441070d4a314fe0915c109e295d5cc2" is_pre_election: true
I20260812 11:14:33.928224 21583 raft_consensus.cc:2454] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c67ebb8931409da653961be51eb68b in term 0.
I20260812 11:14:33.928524 21955 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9; no voters:
I20260812 11:14:33.928838 22185 raft_consensus.cc:2781] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:14:33.928946 22185 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:14:33.929004 22185 raft_consensus.cc:3037] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:33.930168 22185 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.930486 22185 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [CANDIDATE]: Term 1 election: Requested vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457)
I20260812 11:14:33.930822 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc1363dbba44b6b97601406e4bd801" candidate_uuid: "c1c67ebb8931409da653961be51eb68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
I20260812 11:14:33.930841 21583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc1363dbba44b6b97601406e4bd801" candidate_uuid: "c1c67ebb8931409da653961be51eb68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f441070d4a314fe0915c109e295d5cc2"
I20260812 11:14:33.930950 21583 raft_consensus.cc:3037] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:33.930950 21840 raft_consensus.cc:3037] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:14:33.932235 21583 raft_consensus.cc:2454] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c67ebb8931409da653961be51eb68b in term 1.
I20260812 11:14:33.932235 21840 raft_consensus.cc:2454] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c67ebb8931409da653961be51eb68b in term 1.
I20260812 11:14:33.932593 21955 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9; no voters:
I20260812 11:14:33.932768 22185 raft_consensus.cc:2781] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:14:33.932996 22185 raft_consensus.cc:686] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 1 LEADER]: Becoming Leader. State: Replica: c1c67ebb8931409da653961be51eb68b, State: Running, Role: LEADER
I20260812 11:14:33.933333 22185 consensus_queue.cc:237] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:33.936707 21313 catalog_manager.cc:5495] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b reported cstate change: term changed from 0 to 1, leader changed from <none> to c1c67ebb8931409da653961be51eb68b (127.23.97.195). New cstate: current_term: 1 leader_uuid: "c1c67ebb8931409da653961be51eb68b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } health_report { overall_health: UNKNOWN } } }
I20260812 11:14:33.956857 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:33.953397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.23.97.195:39563)
0812 11:14:33.953513 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:33.953516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:33.953551 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:14:33.953705 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:33.953707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:33.953758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:33.953843 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:14:33.954039 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.954545 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.955319 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:33.955520 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:33.955595 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:33.955916 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:14:33.956461 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:14:33.956551 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:33.956611 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 11:14:33.956660 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:14:33.963333 21840 raft_consensus.cc:1261] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Refusing update from remote peer c1c67ebb8931409da653961be51eb68b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:33.963328 21583 raft_consensus.cc:1261] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Refusing update from remote peer c1c67ebb8931409da653961be51eb68b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:14:33.964058 22185 consensus_queue.cc:1035] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:33.964262 22192 consensus_queue.cc:1035] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:14:33.970311 22198 mvcc.cc:205] Tried to move back new op lower bound from 7317640290146189312 to 7317640290033729536. Current Snapshot: MvccSnapshot[applied={T|T < 7317640290146189312}]
I20260812 11:14:33.975556 22200 mvcc.cc:205] Tried to move back new op lower bound from 7317640290146189312 to 7317640290033729536. Current Snapshot: MvccSnapshot[applied={T|T < 7317640290146189312}]
I20260812 11:14:33.980270 21313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:33.980509 21313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:14:33.982975 21313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Sending DeleteTablet for 3 replicas of tablet c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:33.983693 21798 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc1363dbba44b6b97601406e4bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:33 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:34948
I20260812 11:14:33.983924 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc1363dbba44b6b97601406e4bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:33 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:42586
I20260812 11:14:33.984234 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc1363dbba44b6b97601406e4bd801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:14:33 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:47706
I20260812 11:14:33.984318 22216 tablet_replica.cc:330] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: stopping tablet replica
I20260812 11:14:33.984472 22216 raft_consensus.cc:2229] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:33.984632 22216 raft_consensus.cc:2258] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:33.984761 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 21385
I20260812 11:14:33.995692 22215 tablet_replica.cc:330] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: stopping tablet replica
I20260812 11:14:33.996094 22215 raft_consensus.cc:2229] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:14:33.996520 22215 raft_consensus.cc:2258] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:33.996908 22217 tablet_replica.cc:330] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: stopping tablet replica
I20260812 11:14:33.997221 22217 raft_consensus.cc:2229] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:14:33.997586 22217 raft_consensus.cc:2258] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:14:33.999230 22217 ts_tablet_manager.cc:1905] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:33.999558 22215 ts_tablet_manager.cc:1905] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:14:34.002362 22217 ts_tablet_manager.cc:1918] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:34.002362 22215 ts_tablet_manager.cc:1918] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:14:34.002467 22215 log.cc:1196] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:34.002595 22217 log.cc:1196] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:34.002822 22215 ts_tablet_manager.cc:1939] T c3dc1363dbba44b6b97601406e4bd801 P cf950bd8bf6044c1ac91a16fc44e34c9: Deleting consensus metadata
I20260812 11:14:34.003391 22217 ts_tablet_manager.cc:1939] T c3dc1363dbba44b6b97601406e4bd801 P c1c67ebb8931409da653961be51eb68b: Deleting consensus metadata
I20260812 11:14:34.004297 21293 catalog_manager.cc:4841] TS cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): tablet c3dc1363dbba44b6b97601406e4bd801 (table test-table [id=d2cb1a09ef50455eb03414cc68d3dc83]) successfully deleted
I20260812 11:14:34.004561 21293 catalog_manager.cc:4841] TS c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): tablet c3dc1363dbba44b6b97601406e4bd801 (table test-table [id=d2cb1a09ef50455eb03414cc68d3dc83]) successfully deleted
W20260812 11:14:34.004777 21287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44457 (error 108)
I20260812 11:14:34.005956 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:34.005625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44457 (local address 127.0.0.1:42592)
0812 11:14:34.005775 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:34.005863 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 11:14:34.006150 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 21684
W20260812 11:14:34.006198 21287 catalog_manager.cc:4584] TS f441070d4a314fe0915c109e295d5cc2 (127.23.97.193:44457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3dc1363dbba44b6b97601406e4bd801: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44457: connect: Connection refused (error 111)
I20260812 11:14:34.022611 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 21922
I20260812 11:14:34.043820 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 21261
I20260812 11:14:54.058182 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--webserver_interface=127.23.97.254
--webserver_port=39421
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:54.171906 22221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:54.172185 22221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:54.172262 22221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:54.176184 22221 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:14:54.176257 22221 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:14:54.176291 22221 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:14:54.176327 22221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:14:54.176383 22221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:54.176425 22221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:54.176483 22221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:14:54.176520 22221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:14:54.181663 22221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:54.182654 22221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:54.183650 22221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:14:54.192117 22221 server_base.cc:1034] running on GCE node
W20260812 11:14:54.192128 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.192169 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.192255 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.192901 22221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:54.193899 22221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:54.195086 22221 hybrid_clock.cc:648] HybridClock initialized: now 1786533294195061 us; error 37 us; skew 500 ppm
Aug 12 11:14:54 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:54.202656 22221 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:14:54.203552 22221 webserver.cc:460] Webserver started at http://127.23.97.254:39421/ using document root <none> and password file <none>
I20260812 11:14:54.204093 22221 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:54.204185 22221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:54.207850 22221 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:14:54.209954 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.211126 22221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:14:54.211268 22221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "e5374091e05f4834b5876e21165d6dfa"
format_stamp: "Formatted at 2026-08-12 11:14:11 on dist-test-slave-zpvx"
server_key: "098d6be2f93f4dd4f9f892c07de5e4ac"
server_key_iv: "9b6029d3e69381ed9451359c81354a7f"
server_key_version: "encryptionkey@0"
I20260812 11:14:54.211803 22221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:54.229725 22221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:54.232424 22221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:54.232632 22221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:54.239681 22221 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40659
I20260812 11:14:54.239979 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40659 every 8 connection(s)
I20260812 11:14:54.240760 22221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:14:54.244993 22352 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:14:54.245926 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 22221
I20260812 11:14:54.246363 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:44457
--local_ip_for_outbound_sockets=127.23.97.193
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=37631
--webserver_interface=127.23.97.193
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:14:54.246811 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap starting.
I20260812 11:14:54.264146 22352 log.cc:824] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:54.275394 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:14:54.275767 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Bootstrap complete.
I20260812 11:14:54.279796 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:54.280207 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Initialized, Role: FOLLOWER
I20260812 11:14:54.280797 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:54.280946 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:14:54.281036 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:14:54.281133 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:14:54.283000 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:54.283334 22352 leader_election.cc:302] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5374091e05f4834b5876e21165d6dfa; no voters:
I20260812 11:14:54.283677 22352 leader_election.cc:288] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:14:54.283757 22361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:14:54.283967 22361 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [term 3 LEADER]: Becoming Leader. State: Replica: e5374091e05f4834b5876e21165d6dfa, State: Running, Role: LEADER
I20260812 11:14:54.284307 22361 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } }
I20260812 11:14:54.284917 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:14:54.285689 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:54.285840 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:54.286440 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5374091e05f4834b5876e21165d6dfa. Latest consensus state: current_term: 3 leader_uuid: "e5374091e05f4834b5876e21165d6dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5374091e05f4834b5876e21165d6dfa" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40659 } } }
I20260812 11:14:54.286597 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa [sys.catalog]: This master's current role is: LEADER
I20260812 11:14:54.287272 22374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:14:54.292728 22374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:14:54.293393 22374 catalog_manager.cc:1259] Loaded cluster ID: 0428089388c44b5082e4a311ed17f8ee
I20260812 11:14:54.293453 22374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:14:54.296345 22374 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:14:54.297102 22374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Loaded TSK: 0
I20260812 11:14:54.297184 22374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5374091e05f4834b5876e21165d6dfa: Loaded TSK: 1
F20260812 11:14:54.297868 22374 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f54bd8999ad google::LogMessage::Fail() at ??:0
@ 0x7f54bd89c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f54bd8994de google::LogMessage::Flush() at ??:0
@ 0x7f54bd89aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f54c055d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f54c0555f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f54c81fe7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f54c81d899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f54c81d9178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f54c821adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f54c823982b std::function<>::operator()() at ??:0
@ 0x7f54c81d8b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f54c81d9f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f54c81d6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f54c821a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f54bfaeb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f54bfaebc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f54bfaed3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f54bfadc512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f54c11206db start_thread at ??:0
@ 0x7f54bc39761f clone at ??:0
W20260812 11:14:54.402541 22357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:54.402792 22357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:54.402863 22357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:54.407050 22357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:54.407112 22357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:54.407198 22357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:14:54.412997 22357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:44457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=37631
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:54.414204 22357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:54.415292 22357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:54.424769 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.424988 22357 server_base.cc:1034] running on GCE node
W20260812 11:14:54.424810 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.425139 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.425604 22357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:54.426285 22357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:54.427472 22357 hybrid_clock.cc:648] HybridClock initialized: now 1786533294427442 us; error 46 us; skew 500 ppm
Aug 12 11:14:54 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:54.430619 22357 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:14:54.431814 22357 webserver.cc:460] Webserver started at http://127.23.97.193:37631/ using document root <none> and password file <none>
I20260812 11:14:54.432492 22357 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:54.432565 22357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:54.436494 22357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:14:54.438684 22457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.439985 22357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:54.440079 22357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "f441070d4a314fe0915c109e295d5cc2"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "fcbe51fa7b675610ad9febc019ec9996"
server_key_iv: "e72ec4aecf2f0ffdc71848278957ed34"
server_key_version: "encryptionkey@0"
I20260812 11:14:54.440699 22357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:54.463428 22357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:54.466790 22357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:54.467005 22357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:54.467644 22357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:54.468945 22474 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 11:14:54.474900 22357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:14:54.475000 22357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 11:14:54.475085 22357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:14:54.476861 22357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:14:54.476928 22357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:54.477237 22474 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Bootstrap starting.
I20260812 11:14:54.487764 22357 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44457
I20260812 11:14:54.487931 22656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44457 every 8 connection(s)
I20260812 11:14:54.488905 22357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:14:54.494539 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 22357
I20260812 11:14:54.494992 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:35247
--local_ip_for_outbound_sockets=127.23.97.194
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=44383
--webserver_interface=127.23.97.194
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:14:54.499989 22474 log.cc:824] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:14:54.509948 22474 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:14:54.510377 22474 tablet_bootstrap.cc:492] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Bootstrap complete.
I20260812 11:14:54.511899 22474 ts_tablet_manager.cc:1397] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.012s
I20260812 11:14:54.515178 22474 raft_consensus.cc:348] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:54.515731 22474 raft_consensus.cc:729] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f441070d4a314fe0915c109e295d5cc2, State: Initialized, Role: FOLLOWER
I20260812 11:14:54.516306 22474 consensus_queue.cc:260] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:54.517385 22474 ts_tablet_manager.cc:1428] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260812 11:14:54.557497 22664 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:14:54.490946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:43359)
0812 11:14:54.491432 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.491488 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:54.492357 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:54.556548 (+ 64191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
W20260812 11:14:54.558358 22660 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:40659 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:40659: Client connection negotiation failed: client connection to 127.23.97.254:40659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:14:54.559157 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.558827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:42971)
0812 11:14:54.558992 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.559084 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:14:54.559815 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.559619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:44497)
0812 11:14:54.559723 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.559768 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:14:54.560045 22660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:14:54.613762 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:54.614002 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:54.614078 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:54.618021 22665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:54.618070 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:54.618125 22665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:14:54.623086 22665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:35247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=44383
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:54.624023 22665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:54.625056 22665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:54.634684 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.634822 22665 server_base.cc:1034] running on GCE node
W20260812 11:14:54.634663 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.634658 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.635360 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:54.635949 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:54.637133 22665 hybrid_clock.cc:648] HybridClock initialized: now 1786533294637109 us; error 35 us; skew 500 ppm
Aug 12 11:14:54 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:54.639721 22665 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:14:54.640645 22665 webserver.cc:460] Webserver started at http://127.23.97.194:44383/ using document root <none> and password file <none>
I20260812 11:14:54.641180 22665 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:54.641273 22665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:54.644981 22665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:14:54.646973 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.648144 22665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:54.648284 22665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "cf950bd8bf6044c1ac91a16fc44e34c9"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "c3805d29a11f0cb3bf85b0a9d8dfc7b2"
server_key_iv: "bb3281a141c373f8012840b6efc14d84"
server_key_version: "encryptionkey@0"
I20260812 11:14:54.648865 22665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:54.681092 22665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:54.683883 22665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:54.684089 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:54.684851 22665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:54.686043 22665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:54.686131 22665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.686203 22665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:54.686235 22665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.697641 22665 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:35247
I20260812 11:14:54.697679 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:35247 every 8 connection(s)
I20260812 11:14:54.698623 22665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:14:54.702239 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.700684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:55697)
0812 11:14:54.701130 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.701563 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
W20260812 11:14:54.702672 22905 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:40659 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:40659: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
I20260812 11:14:54.703250 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.703037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:49041)
0812 11:14:54.703146 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.703202 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 11:14:54.703819 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.703619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:56083)
0812 11:14:54.703719 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.703764 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 11:14:54.703989 22905 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:14:54.704979 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 22665
I20260812 11:14:54.705353 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:39563
--local_ip_for_outbound_sockets=127.23.97.195
--tserver_master_addrs=127.23.97.254:40659
--webserver_port=42313
--webserver_interface=127.23.97.195
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:14:54.814749 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:14:54.815014 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:14:54.815091 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:14:54.818935 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:14:54.819017 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:14:54.819104 22911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:14:54.824162 22911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:39563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=42313
--tserver_master_addrs=127.23.97.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:14:54.825134 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:14:54.826088 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:14:54.835681 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.835651 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:14:54.835673 22924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:14:54.835882 22911 server_base.cc:1034] running on GCE node
I20260812 11:14:54.836371 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:14:54.836995 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:14:54.838191 22911 hybrid_clock.cc:648] HybridClock initialized: now 1786533294838175 us; error 26 us; skew 500 ppm
Aug 12 11:14:54 dist-test-slave-zpvx krb5kdc[20242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:14:54.840894 22911 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:14:54.841693 22911 webserver.cc:460] Webserver started at http://127.23.97.195:42313/ using document root <none> and password file <none>
I20260812 11:14:54.842249 22911 fs_manager.cc:359] Metadata directory not provided
I20260812 11:14:54.842343 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:14:54.846179 22911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:14:54.848119 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.849318 22911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:14:54.849452 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "c1c67ebb8931409da653961be51eb68b"
format_stamp: "Formatted at 2026-08-12 11:14:12 on dist-test-slave-zpvx"
server_key: "3b51066e072bfc09d89195605dc70c83"
server_key_iv: "c87b03535090ee59a58997e649b0fefe"
server_key_version: "encryptionkey@0"
I20260812 11:14:54.849979 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:14:54.866021 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:14:54.869732 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:14:54.869975 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:14:54.870652 22911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:14:54.871729 22911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:14:54.871804 22911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.871860 22911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:14:54.871886 22911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:14:54.882035 22911 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:39563
I20260812 11:14:54.882066 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:39563 every 8 connection(s)
I20260812 11:14:54.882987 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:14:54.886590 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.885044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:34063)
0812 11:14:54.885486 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.885924 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
W20260812 11:14:54.887109 23163 heartbeater.cc:644] Failed to heartbeat to 127.23.97.254:40659 (0 consecutive failures): Network error: Failed to ping master at 127.23.97.254:40659: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
I20260812 11:14:54.887708 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.887468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:36329)
0812 11:14:54.887602 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.887648 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 11:14:54.888286 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:54.888108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:33355)
0812 11:14:54.888199 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:54.888239 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 11:14:54.888489 23163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:14:54.892347 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 22911
I20260812 11:14:55.561120 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.560651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:42021)
0812 11:14:55.560912 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.560976 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260812 11:14:55.704928 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.704558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:39645)
0812 11:14:55.704777 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.704830 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260812 11:14:55.889458 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.888997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:34787)
0812 11:14:55.889292 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.889350 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260812 11:14:55.922396 23173 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:55.922672 23173 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:55.923791 23173 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
Aug 12 11:14:55 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.195@KRBTEST.COM
Aug 12 11:14:55 dist-test-slave-zpvx krb5kdc[20242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.194@KRBTEST.COM
I20260812 11:14:55.938549 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.924046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35247 (local address 127.23.97.193:50449)
0812 11:14:55.924203 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.924223 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.924346 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.926352 (+ 2006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.926358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.926889 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.927387 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.927405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.929025 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.929031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.929991 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.930002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.930713 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.930938 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.931738 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.931752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.933825 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.935942 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.935945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.935947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.936268 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.936680 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.936687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.936689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.936832 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.937527 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.937536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.937810 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.938007 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.938368 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":140,"spinlock_wait_cycles":4864}
I20260812 11:14:55.938887 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.924420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:39563 (local address 127.23.97.193:43745)
0812 11:14:55.924710 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:55.924727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:14:55.924854 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:14:55.926354 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:14:55.926360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:14:55.926895 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:14:55.927386 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.927404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.929094 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.929097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.929987 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:14:55.929996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.930822 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:14:55.930964 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.931697 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:14:55.931714 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:14:55.933381 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:14:55.935785 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.935791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.935805 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.936268 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.936919 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:14:55.936921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:14:55.936922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:14:55.936979 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:14:55.937625 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:14:55.937628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:14:55.937818 (+ 190us) client_negotiation.cc:769] Sending connection context
0812 11:14:55.938006 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 11:14:55.938727 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":546,"spinlock_wait_cycles":109184,"thread_start_us":94,"threads_started":1}
I20260812 11:14:55.939056 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.924621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:43745 (local address 127.23.97.195:39563)
0812 11:14:55.924927 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.924947 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.924981 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:14:55.925637 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.925643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.925973 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.926260 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.927704 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.928923 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930134 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.930681 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930858 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.933605 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.933612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.933640 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.933674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.935654 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.936442 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.936450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.936461 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.936570 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.937072 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.937075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.937077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.937482 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.937618 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.938449 (+ 831us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.938872 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260812 11:14:55.939337 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:55.924299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:50449 (local address 127.23.97.194:35247)
0812 11:14:55.924849 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 11:14:55.924879 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 11:14:55.924932 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:14:55.925673 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:14:55.925680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:14:55.925985 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:14:55.926346 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:14:55.927682 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.928883 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930141 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:14:55.930584 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:14:55.930713 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:14:55.934013 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:14:55.934018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:14:55.934029 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:14:55.934058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:14:55.935833 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.936433 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.936444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.936448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.936568 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:14:55.936944 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:14:55.936948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:14:55.936949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:14:55.937385 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:14:55.937523 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:14:55.938370 (+ 847us) server_negotiation.cc:299] Negotiation successful
0812 11:14:55.939146 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":233,"threads_started":1}
W20260812 11:14:55.941740 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:14:55.941965 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:55.942022 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:14:55.942364 23173 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:14:56.562418 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.561981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:58957)
0812 11:14:56.562253 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.562320 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260812 11:14:56.706044 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.705626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:47141)
0812 11:14:56.705909 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.705961 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260812 11:14:56.890615 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:56.890221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:33507)
0812 11:14:56.890472 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:56.890523 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260812 11:14:57.563665 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:57.563238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:36787)
0812 11:14:57.563506 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:57.563567 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260812 11:14:57.689558 23185 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:57.689702 23185 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:57.690058 23185 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
W20260812 11:14:57.691049 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:14:57.691262 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:57.691337 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:14:57.691541 23185 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:14:57.707238 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:57.706796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:37655)
0812 11:14:57.707069 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:57.707125 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260812 11:14:57.891796 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:57.891327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:53639)
0812 11:14:57.891605 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:57.891669 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260812 11:14:58.564908 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.564489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:48151)
0812 11:14:58.564750 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:58.564808 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260812 11:14:58.708360 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.707921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:60425)
0812 11:14:58.708216 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:58.708268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260812 11:14:58.893007 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:58.892584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:41991)
0812 11:14:58.892848 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:58.892907 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260812 11:14:59.250329 23197 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:14:59.250465 23197 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:14:59.250933 23197 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
W20260812 11:14:59.252060 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:14:59.252274 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:14:59.252350 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:14:59.252514 23197 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:14:59.566538 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:59.565980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:55031)
0812 11:14:59.566358 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:59.566426 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":105,"threads_started":1}
I20260812 11:14:59.709730 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:59.709320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:33567)
0812 11:14:59.709565 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:59.709625 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260812 11:14:59.894397 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:14:59.893939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:51829)
0812 11:14:59.894236 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:14:59.894298 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260812 11:15:00.567970 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:00.567488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:45919)
0812 11:15:00.567797 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:00.567866 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260812 11:15:00.711067 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:00.710596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:44211)
0812 11:15:00.710901 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:00.710965 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260812 11:15:00.877516 23208 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:00.877667 23208 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:15:00.878181 23208 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
W20260812 11:15:00.879170 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:15:00.879328 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:15:00.879370 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:15:00.879508 23208 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:15:00.895655 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:00.895203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:51183)
0812 11:15:00.895494 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:00.895556 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260812 11:15:01.569550 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:01.569028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:45949)
0812 11:15:01.569373 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:01.569438 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260812 11:15:01.712507 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:01.711907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:35781)
0812 11:15:01.712259 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:01.712329 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260812 11:15:01.897087 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:01.896557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:54605)
0812 11:15:01.896904 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:01.896973 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":80,"threads_started":1}
I20260812 11:15:02.570832 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.570404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:39995)
0812 11:15:02.570674 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:02.570734 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260812 11:15:02.713711 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.713257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:51063)
0812 11:15:02.713541 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:02.713599 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
I20260812 11:15:02.898316 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:02.897936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:52865)
0812 11:15:02.898170 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:02.898225 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260812 11:15:03.572216 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.571748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:56413)
0812 11:15:03.572045 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:03.572105 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260812 11:15:03.715016 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.714574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:46395)
0812 11:15:03.714843 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:03.714904 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260812 11:15:03.801043 23227 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:03.801226 23227 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:15:03.801774 23227 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
W20260812 11:15:03.802629 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:15:03.802812 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:15:03.802893 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:15:03.803033 23227 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:15:03.899559 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:03.899112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:50973)
0812 11:15:03.899399 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:03.899462 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260812 11:15:04.573652 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:04.573133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:45265)
0812 11:15:04.573448 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:04.573525 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260812 11:15:04.716320 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:04.715883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:46831)
0812 11:15:04.716158 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:04.716222 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260812 11:15:04.900915 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:04.900490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:54253)
0812 11:15:04.900751 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:04.900808 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260812 11:15:05.575053 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:05.574556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:50895)
0812 11:15:05.574836 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:05.574896 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260812 11:15:05.717664 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:05.717207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:46025)
0812 11:15:05.717470 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:05.717542 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260812 11:15:05.902266 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:05.901760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:34089)
0812 11:15:05.902076 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:05.902152 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260812 11:15:06.576308 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:06.575868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:60361)
0812 11:15:06.576143 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:06.576200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260812 11:15:06.718902 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:06.718475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:34441)
0812 11:15:06.718750 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:06.718807 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260812 11:15:06.903594 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:06.903120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:56517)
0812 11:15:06.903420 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:06.903490 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260812 11:15:07.301795 23250 raft_consensus.cc:482] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:07.301970 23250 raft_consensus.cc:504] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf950bd8bf6044c1ac91a16fc44e34c9" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35247 } } peers { permanent_uuid: "c1c67ebb8931409da653961be51eb68b" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 39563 } } peers { permanent_uuid: "f441070d4a314fe0915c109e295d5cc2" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44457 } }
I20260812 11:15:07.302549 23250 leader_election.cc:288] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247), c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563)
W20260812 11:15:07.303530 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1c67ebb8931409da653961be51eb68b (127.23.97.195:39563): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
W20260812 11:15:07.303712 22468 leader_election.cc:341] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf950bd8bf6044c1ac91a16fc44e34c9 (127.23.97.194:35247): Not found: Tablet not found: c3dc1363dbba44b6b97601406e4bd801
I20260812 11:15:07.303790 22468 leader_election.cc:302] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f441070d4a314fe0915c109e295d5cc2; no voters: c1c67ebb8931409da653961be51eb68b, cf950bd8bf6044c1ac91a16fc44e34c9
I20260812 11:15:07.303948 23250 raft_consensus.cc:2726] T c3dc1363dbba44b6b97601406e4bd801 P f441070d4a314fe0915c109e295d5cc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:15:07.577538 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.577120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:53757)
0812 11:15:07.577386 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:07.577440 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260812 11:15:07.720106 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.719698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:39119)
0812 11:15:07.719952 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:07.720010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260812 11:15:07.904963 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:07.904504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:35155)
0812 11:15:07.904801 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:07.904860 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260812 11:15:08.578817 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.578370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:55749)
0812 11:15:08.578654 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:08.578717 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260812 11:15:08.721604 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.721069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:33157)
0812 11:15:08.721398 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:08.721478 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260812 11:15:08.906315 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:08.905834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:43905)
0812 11:15:08.906135 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:08.906195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260812 11:15:09.580439 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:09.579876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.193:34491)
0812 11:15:09.580182 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:09.580256 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260812 11:15:09.723172 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:09.722723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.194:37621)
0812 11:15:09.722999 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:09.723068 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260812 11:15:09.904639 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 22357
I20260812 11:15:09.911463 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:09.908717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:40659 (local address 127.23.97.195:51593)
0812 11:15:09.911209 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:09.911307 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.254:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2374,"thread_start_us":2150,"threads_started":1}
I20260812 11:15:09.928187 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 22665
I20260812 11:15:09.941356 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 22911
2026-08-12T11:15:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60259 ms)
[----------] 30 tests from SecurityITest (249648 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:10 dist-test-slave-zpvx krb5kdc[23269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:10 dist-test-slave-zpvx krb5kdc[23269](info): set up 2 sockets
Aug 12 11:15:10 dist-test-slave-zpvx krb5kdc[23269](info): commencing operation
krb5kdc: starting...
W20260812 11:15:12.023265 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.372s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:12.159897 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43367
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:12.273862 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:12.274115 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:12.274154 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:12.278123 23291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:12.278178 23291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:12.278198 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:12.278213 23291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:12.278228 23291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:12.283321 23291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:43367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:12.284264 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:12.285310 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:12.293790 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:12.293788 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:12.293962 23291 server_base.cc:1034] running on GCE node
W20260812 11:15:12.293833 23305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:12.294696 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:12.295794 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:12.296974 23291 hybrid_clock.cc:648] HybridClock initialized: now 1786533312296959 us; error 38 us; skew 500 ppm
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:12.304702 23291 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:12.305655 23291 webserver.cc:460] Webserver started at http://127.23.97.254:39199/ using document root <none> and password file <none>
I20260812 11:15:12.306231 23291 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:12.306329 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:12.306571 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:12.308727 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "84a860798bf54af5bfe57561743e92e8"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "234b6756dd8f9d195b34fe1a304797a1"
server_key_iv: "7a93b3866e4c94e8024e54afb7e0e664"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.309332 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "84a860798bf54af5bfe57561743e92e8"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "234b6756dd8f9d195b34fe1a304797a1"
server_key_iv: "7a93b3866e4c94e8024e54afb7e0e664"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.313109 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:15:12.315529 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.316828 23291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:12.316990 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "84a860798bf54af5bfe57561743e92e8"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "234b6756dd8f9d195b34fe1a304797a1"
server_key_iv: "7a93b3866e4c94e8024e54afb7e0e664"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.317106 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:12.336349 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:12.339248 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:12.339461 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:12.347026 23291 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:43367
I20260812 11:15:12.347043 23422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:43367 every 8 connection(s)
I20260812 11:15:12.347987 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:12.350780 23423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:12.356094 23423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: Bootstrap starting.
I20260812 11:15:12.356817 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 23291
I20260812 11:15:12.356966 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:12.357313 23943 external_mini_cluster.cc:1428] Setting key 09614d7cf7a5b733711ed4301a6dbd8b
I20260812 11:15:12.359082 23423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:12.359812 23423 log.cc:824] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:12.362001 23423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: No bootstrap required, opened a new log
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:12.366541 23423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } }
I20260812 11:15:12.366825 23423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:12.366920 23423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a860798bf54af5bfe57561743e92e8, State: Initialized, Role: FOLLOWER
I20260812 11:15:12.367439 23423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } }
I20260812 11:15:12.367630 23423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:12.367707 23423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:12.367856 23423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:12.369059 23423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } }
I20260812 11:15:12.369446 23423 leader_election.cc:302] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84a860798bf54af5bfe57561743e92e8; no voters:
I20260812 11:15:12.369784 23423 leader_election.cc:288] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:12.369836 23434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:12.370031 23434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [term 1 LEADER]: Becoming Leader. State: Replica: 84a860798bf54af5bfe57561743e92e8, State: Running, Role: LEADER
I20260812 11:15:12.370384 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } }
I20260812 11:15:12.371044 23423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:12.372078 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a860798bf54af5bfe57561743e92e8. Latest consensus state: current_term: 1 leader_uuid: "84a860798bf54af5bfe57561743e92e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } } }
I20260812 11:15:12.372197 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:12.372553 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:12.358577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.23.97.254:43367)
0812 11:15:12.359007 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:12.359023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:12.359057 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:15:12.359709 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:12.359726 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:12.360093 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:12.360499 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:12.361363 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.362410 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.363582 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.364028 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.364138 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:12.367083 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:12.367090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:12.367105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:12.367136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:12.369209 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.369700 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.369706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.369711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.369858 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.370204 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.370208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.370210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.370676 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:12.370893 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:12.371182 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:15:12.371483 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":141,"threads_started":1}
I20260812 11:15:12.372601 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84a860798bf54af5bfe57561743e92e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a860798bf54af5bfe57561743e92e8" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 43367 } } }
I20260812 11:15:12.372876 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8 [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:12.376148 23460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:12.376222 23460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:12.376529 23448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:12.377233 23448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:12.382805 23448 catalog_manager.cc:1347] Generated new cluster ID: 7d39fd20dc494588af586c77d5b61b90
I20260812 11:15:12.382869 23448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:12.392236 23448 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:12.393442 23448 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:12.406399 23448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: Generated new TSK 0
I20260812 11:15:12.407184 23448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:15:12.439893 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:12.553520 23469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:12.553773 23469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:12.553817 23469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:12.557721 23469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:12.557768 23469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:12.557858 23469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:12.562908 23469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:12.563915 23469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:12.565052 23469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:12.574635 23481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:12.574651 23479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:12.574651 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:12.574976 23469 server_base.cc:1034] running on GCE node
I20260812 11:15:12.575363 23469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:12.576002 23469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:12.577190 23469 hybrid_clock.cc:648] HybridClock initialized: now 1786533312577179 us; error 32 us; skew 500 ppm
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:12.579890 23469 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:15:12.580823 23469 webserver.cc:460] Webserver started at http://127.23.97.193:34211/ using document root <none> and password file <none>
I20260812 11:15:12.581398 23469 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:12.581492 23469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:12.581704 23469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:12.583658 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "b297c894543c4884a2f07069d8b3b7b4"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "859ab6368b9aeec3e1bf7909b7edacd7"
server_key_iv: "f3d87cfb557811052b4d1a39145bd7c4"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.584229 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "b297c894543c4884a2f07069d8b3b7b4"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "859ab6368b9aeec3e1bf7909b7edacd7"
server_key_iv: "f3d87cfb557811052b4d1a39145bd7c4"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.587761 23469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:15:12.590145 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.591535 23469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:15:12.591665 23469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "b297c894543c4884a2f07069d8b3b7b4"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "859ab6368b9aeec3e1bf7909b7edacd7"
server_key_iv: "f3d87cfb557811052b4d1a39145bd7c4"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.591774 23469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:12.617611 23469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:12.620779 23469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:12.620981 23469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:12.621592 23469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:12.622426 23469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:12.622498 23469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.622570 23469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:12.622622 23469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.632280 23469 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:41089
I20260812 11:15:12.632304 23718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:41089 every 8 connection(s)
I20260812 11:15:12.633176 23469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:12.636605 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 23469
I20260812 11:15:12.636780 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:12.637109 23943 external_mini_cluster.cc:1428] Setting key afb09c1ca1b0c4e9cb9553239dc786fd
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:15:12.647598 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:12.634968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:48035 (local address 127.23.97.254:43367)
0812 11:15:12.635102 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:12.635106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:12.635861 (+ 755us) server_negotiation.cc:407] Connection header received
0812 11:15:12.637656 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:12.637659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:12.637719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:12.637830 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:12.639150 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.639665 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.641241 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.641476 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.641590 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:12.644001 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:12.644010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:12.644012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:12.644035 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:12.645534 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.646042 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.646046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.646048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.646132 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.646488 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.646492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.646493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.646671 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:12.646787 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:12.647301 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 11:15:12.647425 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 11:15:12.648427 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:12.635281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43367 (local address 127.23.97.193:48035)
0812 11:15:12.635711 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:12.635757 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:12.637420 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:12.638008 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:12.638015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:12.638363 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:12.638987 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:12.639010 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.639795 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.639801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:12.641104 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:12.641114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.641629 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.641633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:12.641797 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:12.642388 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:12.642405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:12.643838 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:12.645653 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:12.645659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:12.645673 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:12.645936 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:12.646235 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:12.646238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:12.646240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:12.646379 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:12.646787 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:12.646795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:12.647027 (+ 232us) client_negotiation.cc:769] Sending connection context
0812 11:15:12.647214 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 11:15:12.647533 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:15:12.649753 23724 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43367
I20260812 11:15:12.650003 23724 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:12.650480 23724 heartbeater.cc:511] Master 127.23.97.254:43367 requested a full tablet report, sending...
I20260812 11:15:12.652179 23352 ts_manager.cc:194] Registered new tserver with Master: b297c894543c4884a2f07069d8b3b7b4 (127.23.97.193:41089)
I20260812 11:15:12.653522 23352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:48035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:15:12.665659 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:12.780444 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:12.780679 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:12.780720 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:12.784744 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:12.784796 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:12.784852 23731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:12.789830 23731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:12.790696 23731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:12.791652 23731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:12.800755 23743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:12.800777 23750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:12.800777 23742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:12.801184 23731 server_base.cc:1034] running on GCE node
I20260812 11:15:12.801594 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:12.802284 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:12.803467 23731 hybrid_clock.cc:648] HybridClock initialized: now 1786533312803451 us; error 27 us; skew 500 ppm
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:12.806219 23731 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:15:12.807114 23731 webserver.cc:460] Webserver started at http://127.23.97.194:33749/ using document root <none> and password file <none>
I20260812 11:15:12.807662 23731 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:12.807751 23731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:12.807991 23731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:12.810046 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "50ab247a8b77439ab4658e4fcc5ee689"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "ff96715a06629e212a12335a4115ecc8"
server_key_iv: "e077b94ce3772e7f928ce9b454cfcd80"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.810612 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "50ab247a8b77439ab4658e4fcc5ee689"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "ff96715a06629e212a12335a4115ecc8"
server_key_iv: "e077b94ce3772e7f928ce9b454cfcd80"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.814325 23731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260812 11:15:12.816793 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.818045 23731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:15:12.818202 23731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "50ab247a8b77439ab4658e4fcc5ee689"
format_stamp: "Formatted at 2026-08-12 11:15:12 on dist-test-slave-zpvx"
server_key: "ff96715a06629e212a12335a4115ecc8"
server_key_iv: "e077b94ce3772e7f928ce9b454cfcd80"
server_key_version: "encryptionkey@0"
I20260812 11:15:12.818313 23731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:12.828457 23731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:12.831259 23731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:12.831453 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:12.832013 23731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:12.832857 23731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:12.832943 23731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.833030 23731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:12.833081 23731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:12.842052 23731 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:45329
I20260812 11:15:12.842087 23972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:45329 every 8 connection(s)
I20260812 11:15:12.842941 23731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
Aug 12 11:15:12 dist-test-slave-zpvx krb5kdc[23269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:12.852317 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 23731
I20260812 11:15:12.852546 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:12.852882 23943 external_mini_cluster.cc:1428] Setting key d5bc5b702c48b40b003819706b3fc6e2
I20260812 11:15:12.856879 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:12.844716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:39035 (local address 127.23.97.254:43367)
0812 11:15:12.844861 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:12.844865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:12.845622 (+ 757us) server_negotiation.cc:407] Connection header received
0812 11:15:12.846939 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:12.846942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:12.847001 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:12.847107 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:12.848249 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.848836 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.850019 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.850244 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.850357 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:12.852977 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:12.852984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:12.852987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:12.853017 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:12.854692 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.855178 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.855184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.855189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.855290 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:12.855624 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:12.855630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:12.855635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:12.855889 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:12.856001 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:12.856530 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 11:15:12.856676 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:15:12.857928 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:12.845041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43367 (local address 127.23.97.194:39035)
0812 11:15:12.845471 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:12.845513 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:12.846718 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:12.847284 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:12.847291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:12.847630 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:12.848091 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:12.848104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.848971 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.848980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:12.849885 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:12.849905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:12.850396 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:12.850400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:12.850569 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:12.851195 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:12.851211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:12.852808 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:12.854811 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:12.854816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:12.854827 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:12.855075 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:12.855400 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:12.855402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:12.855404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:12.855532 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:12.855997 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:12.856001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:12.856218 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 11:15:12.856476 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 11:15:12.856841 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260812 11:15:12.859145 23975 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43367
I20260812 11:15:12.859408 23975 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:12.859953 23975 heartbeater.cc:511] Master 127.23.97.254:43367 requested a full tablet report, sending...
I20260812 11:15:12.861090 23352 ts_manager.cc:194] Registered new tserver with Master: 50ab247a8b77439ab4658e4fcc5ee689 (127.23.97.194:45329)
I20260812 11:15:12.861956 23352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:39035
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:15:12.883467 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:12.993237 23984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:12.993462 23984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:12.993502 23984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:12.997737 23984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:12.997793 23984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:12.997849 23984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:13.003059 23984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:43367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:13.003958 23984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.004930 23984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:13.014400 23994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.014393 24000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.014623 23984 server_base.cc:1034] running on GCE node
W20260812 11:15:13.014393 23995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.015118 23984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.015723 23984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.016916 23984 hybrid_clock.cc:648] HybridClock initialized: now 1786533313016903 us; error 25 us; skew 500 ppm
Aug 12 11:15:13 dist-test-slave-zpvx krb5kdc[23269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:13.019737 23984 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:15:13.020864 23984 webserver.cc:460] Webserver started at http://127.23.97.195:43657/ using document root <none> and password file <none>
I20260812 11:15:13.021530 23984 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.021623 23984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.021914 23984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.024006 23984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "f007cfecf8ba4f3c869b9172bf88abad"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "6beaf0f19ba35e0b8e33cf019d96eeab"
server_key_iv: "4bab977302499e9c3a920a6705ba3d9c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.024631 23984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "f007cfecf8ba4f3c869b9172bf88abad"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "6beaf0f19ba35e0b8e33cf019d96eeab"
server_key_iv: "4bab977302499e9c3a920a6705ba3d9c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.028182 23984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:13.030524 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.031733 23984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:13.031876 23984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "f007cfecf8ba4f3c869b9172bf88abad"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "6beaf0f19ba35e0b8e33cf019d96eeab"
server_key_iv: "4bab977302499e9c3a920a6705ba3d9c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.031985 23984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.048971 23984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.051713 23984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.051911 23984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.052521 23984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:13.053347 23984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:13.053421 23984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.053493 23984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:13.053534 23984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.062595 23984 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:40763
I20260812 11:15:13.062623 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:40763 every 8 connection(s)
I20260812 11:15:13.063457 23984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:13.069752 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 23984
I20260812 11:15:13.069922 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:13.070207 23943 external_mini_cluster.cc:1428] Setting key 41c0dadbb1897421a419e52bb7bcc481
Aug 12 11:15:13 dist-test-slave-zpvx krb5kdc[23269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:13.077271 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.065120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49027 (local address 127.23.97.254:43367)
0812 11:15:13.065267 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.065271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.066055 (+ 784us) server_negotiation.cc:407] Connection header received
0812 11:15:13.067435 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.067438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.067497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.067601 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.068787 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.069330 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.070600 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.070909 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.071034 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.073869 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.073873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.073880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:13.073905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.075162 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.075679 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.075684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.075686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.075769 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.076139 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.076143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.076145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.076355 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.076512 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.076984 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.077104 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:15:13.078011 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.065467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:43367 (local address 127.23.97.195:49027)
0812 11:15:13.065907 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.065949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.067213 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.067771 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.067778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.068111 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:13.068640 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.068654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.069469 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.069473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.070472 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.070483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.071203 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.071208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.071359 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.071955 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:13.071971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:13.073691 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:13.075293 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.075299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.075314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.075567 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.075868 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:13.075871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:13.075873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:13.076013 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:13.076511 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:13.076517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:13.076747 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.076941 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.077242 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260812 11:15:13.079079 24232 heartbeater.cc:347] Connected to a master server at 127.23.97.254:43367
I20260812 11:15:13.079284 24232 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:13.079679 24232 heartbeater.cc:511] Master 127.23.97.254:43367 requested a full tablet report, sending...
I20260812 11:15:13.080647 23352 ts_manager.cc:194] Registered new tserver with Master: f007cfecf8ba4f3c869b9172bf88abad (127.23.97.195:40763)
I20260812 11:15:13.081142 23352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:49027
I20260812 11:15:13.083930 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:13.093250 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.085937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.23.97.254:43367)
0812 11:15:13.086070 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.086075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.086253 (+ 178us) server_negotiation.cc:407] Connection header received
0812 11:15:13.086417 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.086420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.086456 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.086534 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.087229 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.087715 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.088657 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.088871 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.088972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.090248 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.090251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.090254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:13.090284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.091483 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.091880 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.091882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.091883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.091977 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:13.092271 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:13.092276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:13.092277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:13.092548 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.092690 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.092965 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.093091 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:15:13.095842 23352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:13.098826 23352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:13.112792 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.107065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.23.97.194:45329)
0812 11:15:13.107417 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.107430 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.107762 (+ 332us) server_negotiation.cc:407] Connection header received
0812 11:15:13.107993 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.107998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.108218 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.108790 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:13.108951 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.109937 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.111405 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.112059 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.112192 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.112356 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.112551 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.112651 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":161,"threads_started":1}
I20260812 11:15:13.112782 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.106898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45329 (local address 127.0.0.1:36820)
0812 11:15:13.107615 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.107669 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.107809 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.108453 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.108459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.108482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:13.108630 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.108639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.110072 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.110077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.111053 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.111068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.112194 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.112197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.112307 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.112347 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.112508 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.112639 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":72,"threads_started":1}
I20260812 11:15:13.113921 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.107641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41089 (local address 127.0.0.1:38838)
0812 11:15:13.108302 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.108331 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.108457 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.109225 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.109228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.109249 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:13.109365 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.109371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.110957 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.110961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.112412 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.112418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.113533 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.113537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.113623 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.113646 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.113719 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.113787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":125,"thread_start_us":96,"threads_started":1}
I20260812 11:15:13.113950 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.108182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.23.97.193:41089)
0812 11:15:13.108725 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.108745 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.108766 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:15:13.108838 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.108845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.109045 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.109226 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:13.109508 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.110774 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.112548 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.113400 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.113515 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.113653 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.113762 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.113837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":122,"threads_started":1}
I20260812 11:15:13.114526 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.106932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:40763 (local address 127.0.0.1:55392)
0812 11:15:13.107885 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.107899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.107988 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.109408 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.109411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.109429 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:13.109528 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.109533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.110957 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.110961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.112401 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.112412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.114090 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.114096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.114204 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.114233 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.114323 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.114402 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":118,"thread_start_us":97,"threads_started":1}
I20260812 11:15:13.114948 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.108238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.23.97.195:40763)
0812 11:15:13.108917 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.108926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.108944 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:13.109003 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.109007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.109195 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.109346 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:13.109658 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.110797 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.112547 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.113399 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.113515 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.113653 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.114691 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.114800 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":167,"threads_started":1}
I20260812 11:15:13.116331 23838 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 (DEFAULT_TABLE table=test-table [id=2b52996a6cff431e84317dd99206bb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:13.116798 23594 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 (DEFAULT_TABLE table=test-table [id=2b52996a6cff431e84317dd99206bb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:13.117547 23838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:13.117339 24110 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 (DEFAULT_TABLE table=test-table [id=2b52996a6cff431e84317dd99206bb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:13.117786 23594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:13.118359 24110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:13.125514 24279 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Bootstrap starting.
I20260812 11:15:13.126040 24286 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: Bootstrap starting.
I20260812 11:15:13.127619 24279 tablet_bootstrap.cc:654] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:13.127750 24286 tablet_bootstrap.cc:654] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:13.128947 24286 log.cc:824] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:13.128950 24279 log.cc:824] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:13.129355 24282 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Bootstrap starting.
I20260812 11:15:13.131139 24286 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: No bootstrap required, opened a new log
I20260812 11:15:13.131156 24279 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: No bootstrap required, opened a new log
I20260812 11:15:13.131357 24286 ts_tablet_manager.cc:1397] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:15:13.131361 24279 ts_tablet_manager.cc:1397] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:13.131441 24282 tablet_bootstrap.cc:654] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:13.132747 24282 log.cc:824] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:13.134720 24282 tablet_bootstrap.cc:492] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: No bootstrap required, opened a new log
I20260812 11:15:13.134905 24282 ts_tablet_manager.cc:1397] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:13.134709 24286 raft_consensus.cc:348] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.135018 24286 raft_consensus.cc:374] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:13.135111 24286 raft_consensus.cc:729] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b297c894543c4884a2f07069d8b3b7b4, State: Initialized, Role: FOLLOWER
I20260812 11:15:13.135254 24279 raft_consensus.cc:348] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.135815 24279 raft_consensus.cc:374] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:13.135887 24279 raft_consensus.cc:729] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f007cfecf8ba4f3c869b9172bf88abad, State: Initialized, Role: FOLLOWER
I20260812 11:15:13.136147 24286 consensus_queue.cc:260] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.136343 24279 consensus_queue.cc:260] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.137140 24286 ts_tablet_manager.cc:1428] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:13.137308 24279 ts_tablet_manager.cc:1428] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 11:15:13.137537 23724 heartbeater.cc:503] Master 127.23.97.254:43367 was elected leader, sending a full tablet report...
I20260812 11:15:13.137666 24232 heartbeater.cc:503] Master 127.23.97.254:43367 was elected leader, sending a full tablet report...
I20260812 11:15:13.139947 24282 raft_consensus.cc:348] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.140228 24282 raft_consensus.cc:374] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:13.140280 24282 raft_consensus.cc:729] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ab247a8b77439ab4658e4fcc5ee689, State: Initialized, Role: FOLLOWER
I20260812 11:15:13.140880 24282 consensus_queue.cc:260] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.141515 23975 heartbeater.cc:503] Master 127.23.97.254:43367 was elected leader, sending a full tablet report...
I20260812 11:15:13.141783 24282 ts_tablet_manager.cc:1428] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:15:13.152493 24296 raft_consensus.cc:482] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:13.152683 24296 raft_consensus.cc:504] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.153690 24296 leader_election.cc:288] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f007cfecf8ba4f3c869b9172bf88abad (127.23.97.195:40763), b297c894543c4884a2f07069d8b3b7b4 (127.23.97.193:41089)
I20260812 11:15:13.158485 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.154031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:53663 (local address 127.23.97.195:40763)
0812 11:15:13.154151 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.154156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.154172 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:13.154236 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.154242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.154309 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.154389 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:13.154846 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.155620 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157124 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.157775 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157917 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.157970 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.158284 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.158377 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:15:13.158485 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.154318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:56581 (local address 127.23.97.193:41089)
0812 11:15:13.154436 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.154443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.154502 (+ 59us) server_negotiation.cc:407] Connection header received
0812 11:15:13.154674 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.154680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.154735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.154833 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:13.155119 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.155832 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157211 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.157794 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157917 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.157969 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.158284 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.158359 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:15:13.158502 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.153930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:40763 (local address 127.23.97.194:53663)
0812 11:15:13.154030 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.154048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.154140 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.154515 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.154520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.154542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:13.154717 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.154727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.155732 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.155734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.156984 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.156996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.157922 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.158147 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.158162 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.158246 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.158338 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":86}
I20260812 11:15:13.158511 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.153930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41089 (local address 127.23.97.194:56581)
0812 11:15:13.154411 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.154425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.154555 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.154867 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.154870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.154890 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:13.155003 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.155008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.155928 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.155931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.157084 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.157092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.157922 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.157926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.158122 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.158162 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.158246 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.158338 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":158,"thread_start_us":106,"threads_started":1}
I20260812 11:15:13.159361 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9bb61a1b4b4a70a4bfe1f0db0c82c5" candidate_uuid: "50ab247a8b77439ab4658e4fcc5ee689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b297c894543c4884a2f07069d8b3b7b4" is_pre_election: true
I20260812 11:15:13.159363 24141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9bb61a1b4b4a70a4bfe1f0db0c82c5" candidate_uuid: "50ab247a8b77439ab4658e4fcc5ee689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f007cfecf8ba4f3c869b9172bf88abad" is_pre_election: true
I20260812 11:15:13.159646 23634 raft_consensus.cc:2454] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ab247a8b77439ab4658e4fcc5ee689 in term 0.
I20260812 11:15:13.159663 24141 raft_consensus.cc:2454] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ab247a8b77439ab4658e4fcc5ee689 in term 0.
I20260812 11:15:13.160126 23764 leader_election.cc:302] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ab247a8b77439ab4658e4fcc5ee689, b297c894543c4884a2f07069d8b3b7b4; no voters:
I20260812 11:15:13.160422 24296 raft_consensus.cc:2781] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:13.160511 24296 raft_consensus.cc:482] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:13.160557 24296 raft_consensus.cc:3037] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:13.161861 24296 raft_consensus.cc:504] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.162191 24296 leader_election.cc:288] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [CANDIDATE]: Term 1 election: Requested vote from peers f007cfecf8ba4f3c869b9172bf88abad (127.23.97.195:40763), b297c894543c4884a2f07069d8b3b7b4 (127.23.97.193:41089)
I20260812 11:15:13.162523 24141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9bb61a1b4b4a70a4bfe1f0db0c82c5" candidate_uuid: "50ab247a8b77439ab4658e4fcc5ee689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f007cfecf8ba4f3c869b9172bf88abad"
I20260812 11:15:13.162526 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9bb61a1b4b4a70a4bfe1f0db0c82c5" candidate_uuid: "50ab247a8b77439ab4658e4fcc5ee689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b297c894543c4884a2f07069d8b3b7b4"
I20260812 11:15:13.162657 24141 raft_consensus.cc:3037] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:13.162669 23634 raft_consensus.cc:3037] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:13.163676 23634 raft_consensus.cc:2454] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ab247a8b77439ab4658e4fcc5ee689 in term 1.
I20260812 11:15:13.163810 24141 raft_consensus.cc:2454] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ab247a8b77439ab4658e4fcc5ee689 in term 1.
I20260812 11:15:13.163968 23764 leader_election.cc:302] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ab247a8b77439ab4658e4fcc5ee689, b297c894543c4884a2f07069d8b3b7b4; no voters:
I20260812 11:15:13.164148 24296 raft_consensus.cc:2781] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:13.164403 24296 raft_consensus.cc:686] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 1 LEADER]: Becoming Leader. State: Replica: 50ab247a8b77439ab4658e4fcc5ee689, State: Running, Role: LEADER
I20260812 11:15:13.164736 24296 consensus_queue.cc:237] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } }
I20260812 11:15:13.167881 23352 catalog_manager.cc:5495] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50ab247a8b77439ab4658e4fcc5ee689 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "50ab247a8b77439ab4658e4fcc5ee689" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ab247a8b77439ab4658e4fcc5ee689" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:13.190917 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.187682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.23.97.194:45329)
0812 11:15:13.187813 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.187816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.187828 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:13.187922 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.187924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.187974 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.188060 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:13.188277 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.188830 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.189670 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.189878 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.189956 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.190210 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:13.190585 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:13.190673 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.190739 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.190785 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:15:13.196739 24141 raft_consensus.cc:1261] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 1 FOLLOWER]: Refusing update from remote peer 50ab247a8b77439ab4658e4fcc5ee689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:13.196739 23634 raft_consensus.cc:1261] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 1 FOLLOWER]: Refusing update from remote peer 50ab247a8b77439ab4658e4fcc5ee689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:13.197511 24296 consensus_queue.cc:1035] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b297c894543c4884a2f07069d8b3b7b4" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:13.197779 24299 consensus_queue.cc:1035] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f007cfecf8ba4f3c869b9172bf88abad" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:13.205286 24304 mvcc.cc:205] Tried to move back new op lower bound from 7317640450846248960 to 7317640450725224448. Current Snapshot: MvccSnapshot[applied={T|T < 7317640450846248960}]
I20260812 11:15:13.206099 24305 mvcc.cc:205] Tried to move back new op lower bound from 7317640450846248960 to 7317640450725224448. Current Snapshot: MvccSnapshot[applied={T|T < 7317640450846248960}]
I20260812 11:15:13.214568 23352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:13.214746 23352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:13.217252 23352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84a860798bf54af5bfe57561743e92e8: Sending DeleteTablet for 3 replicas of tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5
I20260812 11:15:13.218029 23594 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:13 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:38838
I20260812 11:15:13.218005 23838 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:13 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:36820
I20260812 11:15:13.218503 24321 tablet_replica.cc:330] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: stopping tablet replica
I20260812 11:15:13.218500 24110 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:13 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:55392
I20260812 11:15:13.218916 24321 raft_consensus.cc:2229] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:13.218984 24322 tablet_replica.cc:330] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4: stopping tablet replica
I20260812 11:15:13.219288 24322 raft_consensus.cc:2229] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P b297c894543c4884a2f07069d8b3b7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:13.219341 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 23469
I20260812 11:15:13.219337 24321 raft_consensus.cc:2258] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:13.231901 24323 tablet_replica.cc:330] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: stopping tablet replica
I20260812 11:15:13.232271 24323 raft_consensus.cc:2229] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:13.232654 24323 raft_consensus.cc:2258] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:13.233673 24321 ts_tablet_manager.cc:1905] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:13.234133 24323 ts_tablet_manager.cc:1905] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:13.237135 24323 ts_tablet_manager.cc:1918] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:13.237243 24323 log.cc:1196] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/1c9bb61a1b4b4a70a4bfe1f0db0c82c5
I20260812 11:15:13.237618 24323 ts_tablet_manager.cc:1939] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P f007cfecf8ba4f3c869b9172bf88abad: Deleting consensus metadata
I20260812 11:15:13.237816 24321 ts_tablet_manager.cc:1918] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:13.237979 24321 log.cc:1196] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/1c9bb61a1b4b4a70a4bfe1f0db0c82c5
I20260812 11:15:13.238399 24321 ts_tablet_manager.cc:1939] T 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 P 50ab247a8b77439ab4658e4fcc5ee689: Deleting consensus metadata
I20260812 11:15:13.238974 23321 catalog_manager.cc:4841] TS f007cfecf8ba4f3c869b9172bf88abad (127.23.97.195:40763): tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 (table test-table [id=2b52996a6cff431e84317dd99206bb71]) successfully deleted
W20260812 11:15:13.239238 23326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:41089 (error 108)
I20260812 11:15:13.239915 23325 catalog_manager.cc:4841] TS 50ab247a8b77439ab4658e4fcc5ee689 (127.23.97.194:45329): tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5 (table test-table [id=2b52996a6cff431e84317dd99206bb71]) successfully deleted
I20260812 11:15:13.240352 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.240026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:41089 (local address 127.0.0.1:38844)
0812 11:15:13.240170 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.240249 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 11:15:13.240499 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 23731
W20260812 11:15:13.240533 23326 catalog_manager.cc:4584] TS b297c894543c4884a2f07069d8b3b7b4 (127.23.97.193:41089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c9bb61a1b4b4a70a4bfe1f0db0c82c5: Network error: Client connection negotiation failed: client connection to 127.23.97.193:41089: connect: Connection refused (error 111)
I20260812 11:15:13.264606 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 23984
I20260812 11:15:13.287334 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 23291
2026-08-12T11:15:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T11:15:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:13Z Disabled control of system clock
I20260812 11:15:13.325660 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46261
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:13.435768 24330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.436015 24330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.436050 24330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.439993 24330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:13.440042 24330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.440058 24330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.440081 24330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:13.440146 24330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:13.444766 24330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:13.445767 24330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.447180 24330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:13.455757 24343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.455770 24348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.455734 24339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.455957 24330 server_base.cc:1034] running on GCE node
I20260812 11:15:13.456617 24330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.457718 24330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.458909 24330 hybrid_clock.cc:648] HybridClock initialized: now 1786533313458883 us; error 50 us; skew 500 ppm
I20260812 11:15:13.460918 24330 webserver.cc:460] Webserver started at http://127.23.97.254:40917/ using document root <none> and password file <none>
I20260812 11:15:13.461503 24330 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.461599 24330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.461843 24330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.463888 24330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "c07a3b732dba4ca092eb52ffa5539904"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "fe4995e3b28c45ca1bdf3a2b50ac4d26"
server_key_iv: "fcf0261e4e897a07d4fd79371487256c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.464546 24330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "c07a3b732dba4ca092eb52ffa5539904"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "fe4995e3b28c45ca1bdf3a2b50ac4d26"
server_key_iv: "fcf0261e4e897a07d4fd79371487256c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.468158 24330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:13.470687 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.471918 24330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:13.472074 24330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c07a3b732dba4ca092eb52ffa5539904"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "fe4995e3b28c45ca1bdf3a2b50ac4d26"
server_key_iv: "fcf0261e4e897a07d4fd79371487256c"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.472189 24330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.480854 24330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.481485 24330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.481678 24330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.488792 24330 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:46261
I20260812 11:15:13.488822 24470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:46261 every 8 connection(s)
I20260812 11:15:13.489727 24330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:13.491415 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24330
I20260812 11:15:13.491559 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:13.491917 23943 external_mini_cluster.cc:1428] Setting key d463bfc998a66fe031f510017a86670c
I20260812 11:15:13.493474 24473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:13.498876 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: Bootstrap starting.
I20260812 11:15:13.501137 24473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:13.501447 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.493381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.23.97.254:46261)
0812 11:15:13.494153 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.494170 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.494205 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:13.494837 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.494853 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.495238 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.495574 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.495994 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.497040 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.498009 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.498483 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.498698 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.499600 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.499605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.499617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:13.499993 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.500049 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:13.500079 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.500199 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.500326 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.500620 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":153,"threads_started":1}
I20260812 11:15:13.502084 24473 log.cc:824] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:13.504177 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: No bootstrap required, opened a new log
I20260812 11:15:13.507704 24473 raft_consensus.cc:348] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } }
I20260812 11:15:13.507906 24473 raft_consensus.cc:374] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:13.507997 24473 raft_consensus.cc:729] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07a3b732dba4ca092eb52ffa5539904, State: Initialized, Role: FOLLOWER
I20260812 11:15:13.508517 24473 consensus_queue.cc:260] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } }
I20260812 11:15:13.508664 24473 raft_consensus.cc:388] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:13.508730 24473 raft_consensus.cc:482] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:13.508862 24473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:13.509718 24473 raft_consensus.cc:504] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } }
I20260812 11:15:13.510056 24473 leader_election.cc:302] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c07a3b732dba4ca092eb52ffa5539904; no voters:
I20260812 11:15:13.510375 24473 leader_election.cc:288] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:13.510614 24486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:13.510860 24486 raft_consensus.cc:686] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [term 1 LEADER]: Becoming Leader. State: Replica: c07a3b732dba4ca092eb52ffa5539904, State: Running, Role: LEADER
I20260812 11:15:13.511211 24486 consensus_queue.cc:237] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } }
I20260812 11:15:13.511751 24473 sys_catalog.cc:564] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:13.513043 24487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c07a3b732dba4ca092eb52ffa5539904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } } }
I20260812 11:15:13.513064 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c07a3b732dba4ca092eb52ffa5539904. Latest consensus state: current_term: 1 leader_uuid: "c07a3b732dba4ca092eb52ffa5539904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c07a3b732dba4ca092eb52ffa5539904" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46261 } } }
I20260812 11:15:13.513187 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:13.513186 24487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:13.513657 24503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:13.516911 24503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:13.521754 24503 catalog_manager.cc:1347] Generated new cluster ID: f91fcf7b5ebd473ea5cb07b7addd4180
I20260812 11:15:13.521816 24503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:13.539783 24503 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:13.541113 24503 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:13.553113 24503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: Generated new TSK 0
I20260812 11:15:13.553905 24503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:13.562902 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--builtin_ntp_servers=127.23.97.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:13.677518 24523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.677855 24523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.677923 24523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.682291 24523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.682345 24523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.682430 24523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:13.687175 24523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:13.688206 24523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.689653 24523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:13.699177 24532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.699152 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.699148 24536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.699429 24523 server_base.cc:1034] running on GCE node
I20260812 11:15:13.699900 24523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.700568 24523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.701725 24523 hybrid_clock.cc:648] HybridClock initialized: now 1786533313701713 us; error 26 us; skew 500 ppm
I20260812 11:15:13.703644 24523 webserver.cc:460] Webserver started at http://127.23.97.193:44309/ using document root <none> and password file <none>
I20260812 11:15:13.704164 24523 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.704221 24523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.704453 24523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.706427 24523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "d78dc924742146beba8903f3b3edbf03"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "0c3c51706d81eb7acca96439cce06163"
server_key_iv: "8c06b6c9b3d5a8a0b17c2aea7bbad187"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.706931 24523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "d78dc924742146beba8903f3b3edbf03"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "0c3c51706d81eb7acca96439cce06163"
server_key_iv: "8c06b6c9b3d5a8a0b17c2aea7bbad187"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.710618 24523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 11:15:13.713164 24553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.714414 24523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:13.714519 24523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "d78dc924742146beba8903f3b3edbf03"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "0c3c51706d81eb7acca96439cce06163"
server_key_iv: "8c06b6c9b3d5a8a0b17c2aea7bbad187"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.714601 24523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.744145 24523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.744856 24523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.745013 24523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.745574 24523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:13.746507 24523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:13.746556 24523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.746596 24523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:13.746618 24523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.755563 24523 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:33457
I20260812 11:15:13.755591 24770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:33457 every 8 connection(s)
I20260812 11:15:13.756502 24523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:13.760428 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24523
I20260812 11:15:13.760572 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:13.760926 23943 external_mini_cluster.cc:1428] Setting key 26167b5a47abc150e6834e13e6ca4b49
I20260812 11:15:13.763197 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--builtin_ntp_servers=127.23.97.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:13.767439 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.758454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:41387 (local address 127.23.97.254:46261)
0812 11:15:13.758588 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.758591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.759780 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 11:15:13.761757 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.761760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.761837 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.761927 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.763070 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.763704 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.765449 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.765693 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.765810 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.766926 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.766930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.766933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:13.766953 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.766973 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:13.766980 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.767059 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.767218 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.767286 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:15:13.768359 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.758942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46261 (local address 127.23.97.193:41387)
0812 11:15:13.759613 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.759655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.761500 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.762090 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.762097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.762182 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:13.762906 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.762930 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.763846 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.763852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.765317 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.765328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.765861 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.765866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.766026 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.766654 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:13.766655 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:13.766697 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:13.766760 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:13.767062 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:13.767075 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:13.767082 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.767217 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.767499 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260812 11:15:13.769850 24772 heartbeater.cc:347] Connected to a master server at 127.23.97.254:46261
I20260812 11:15:13.770088 24772 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:13.770601 24772 heartbeater.cc:511] Master 127.23.97.254:46261 requested a full tablet report, sending...
I20260812 11:15:13.772408 24406 ts_manager.cc:194] Registered new tserver with Master: d78dc924742146beba8903f3b3edbf03 (127.23.97.193:33457)
I20260812 11:15:13.774181 24406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.193:41387
W20260812 11:15:13.876792 24782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:13.877038 24782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:13.877074 24782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:13.881053 24782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:13.881100 24782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:13.881152 24782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:13.885674 24782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:13.886617 24782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:13.887881 24782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:13.897394 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.897388 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:13.897387 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:13.897729 24782 server_base.cc:1034] running on GCE node
I20260812 11:15:13.898099 24782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:13.898697 24782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:13.899885 24782 hybrid_clock.cc:648] HybridClock initialized: now 1786533313899872 us; error 28 us; skew 500 ppm
I20260812 11:15:13.901633 24782 webserver.cc:460] Webserver started at http://127.23.97.194:34697/ using document root <none> and password file <none>
I20260812 11:15:13.902199 24782 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:13.902300 24782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:13.902532 24782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:13.904748 24782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "5aec3655515d46b8af3912d610d8345c"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "e143e014a7ce96eef8586f00773f4cf9"
server_key_iv: "fc3bbcdd824e6016d40356bf84653fe2"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.905313 24782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "5aec3655515d46b8af3912d610d8345c"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "e143e014a7ce96eef8586f00773f4cf9"
server_key_iv: "fc3bbcdd824e6016d40356bf84653fe2"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.908799 24782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:15:13.911109 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.912241 24782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:13.912415 24782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "5aec3655515d46b8af3912d610d8345c"
format_stamp: "Formatted at 2026-08-12 11:15:13 on dist-test-slave-zpvx"
server_key: "e143e014a7ce96eef8586f00773f4cf9"
server_key_iv: "fc3bbcdd824e6016d40356bf84653fe2"
server_key_version: "encryptionkey@0"
I20260812 11:15:13.912536 24782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:13.930621 24782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:13.931246 24782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:13.931442 24782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:13.931994 24782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:13.932904 24782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:13.932977 24782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.933050 24782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:13.933094 24782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:13.941820 24782 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:35273
I20260812 11:15:13.941857 25017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:35273 every 8 connection(s)
I20260812 11:15:13.942675 24782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:13.950554 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 24782
I20260812 11:15:13.950699 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:13.951061 23943 external_mini_cluster.cc:1428] Setting key cb69ca3e8de4bcc4d272452a5d1566d3
I20260812 11:15:13.952119 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.944412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:44761 (local address 127.23.97.254:46261)
0812 11:15:13.944548 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:13.944552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:13.945552 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 11:15:13.946979 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:13.946983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:13.947048 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:13.947158 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:13.948035 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.948583 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.949768 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.950058 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.950164 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.951603 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:13.951607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:13.951609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:13.951623 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:13.951647 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:13.951675 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:13.951749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:13.951954 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 11:15:13.952009 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:15:13.953125 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:13.944765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46261 (local address 127.23.97.194:44761)
0812 11:15:13.945404 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:13.945443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:13.946751 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:13.947339 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:13.947346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:13.947427 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:13.947888 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.947901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.948729 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.948732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.949612 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:13.949622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:13.950281 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:13.950290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:13.950573 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:13.951310 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:13.951312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:13.951365 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:13.951441 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:13.951779 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:13.951788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:13.951793 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:15:13.951942 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:15:13.952205 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260812 11:15:13.953277 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--builtin_ntp_servers=127.23.97.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:13.954249 25018 heartbeater.cc:347] Connected to a master server at 127.23.97.254:46261
I20260812 11:15:13.954497 25018 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:13.955116 25018 heartbeater.cc:511] Master 127.23.97.254:46261 requested a full tablet report, sending...
I20260812 11:15:13.956207 24406 ts_manager.cc:194] Registered new tserver with Master: 5aec3655515d46b8af3912d610d8345c (127.23.97.194:35273)
I20260812 11:15:13.956784 24406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.194:44761
W20260812 11:15:14.068174 25022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:14.068511 25022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:14.068611 25022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:14.072573 25022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:14.072639 25022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:14.072729 25022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:14.077404 25022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:14.078394 25022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:14.079742 25022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:14.089169 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:14.089169 25030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:14.089339 25022 server_base.cc:1034] running on GCE node
W20260812 11:15:14.089499 25035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:14.089871 25022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:14.090461 25022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:14.091622 25022 hybrid_clock.cc:648] HybridClock initialized: now 1786533314091607 us; error 27 us; skew 500 ppm
I20260812 11:15:14.093519 25022 webserver.cc:460] Webserver started at http://127.23.97.195:45081/ using document root <none> and password file <none>
I20260812 11:15:14.094084 25022 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:14.094146 25022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:14.094358 25022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:14.096509 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "28cc5378c8e04a26bf94e64c5e3951bf"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-zpvx"
server_key: "747b662ea00960469842e68e1e32b56c"
server_key_iv: "1de40068cf018a22e39be96b908d2738"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.097086 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "28cc5378c8e04a26bf94e64c5e3951bf"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-zpvx"
server_key: "747b662ea00960469842e68e1e32b56c"
server_key_iv: "1de40068cf018a22e39be96b908d2738"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.100773 25022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:15:14.103073 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.104476 25022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:14.104576 25022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "28cc5378c8e04a26bf94e64c5e3951bf"
format_stamp: "Formatted at 2026-08-12 11:15:14 on dist-test-slave-zpvx"
server_key: "747b662ea00960469842e68e1e32b56c"
server_key_iv: "1de40068cf018a22e39be96b908d2738"
server_key_version: "encryptionkey@0"
I20260812 11:15:14.104656 25022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:14.124342 25022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:14.125061 25022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:14.125216 25022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:14.125768 25022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:14.126951 25022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:14.127013 25022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.127066 25022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:14.127090 25022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:14.136206 25022 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:34273
I20260812 11:15:14.136245 25282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:34273 every 8 connection(s)
I20260812 11:15:14.137152 25022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:14.140789 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25022
I20260812 11:15:14.140916 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:14.141222 23943 external_mini_cluster.cc:1428] Setting key 5e514c048a234a6cb268cca434189f46
I20260812 11:15:14.146862 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.139065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:52187 (local address 127.23.97.254:46261)
0812 11:15:14.139231 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.139236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.140383 (+ 1147us) server_negotiation.cc:407] Connection header received
0812 11:15:14.142025 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.142028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.142083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.142170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:14.143103 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.143737 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.144955 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.145178 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.145299 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.146344 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:14.146348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:14.146351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:14.146364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:14.146391 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:14.146399 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:14.146473 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.146647 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.146709 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:15:14.147684 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.139533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46261 (local address 127.23.97.195:52187)
0812 11:15:14.140208 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.140251 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.141796 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.142368 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.142375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.142462 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:14.142957 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.142971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.143871 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.143875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.144827 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.144837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.145336 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.145341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.145514 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.146082 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:14.146084 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:14.146124 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:14.146187 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:14.146488 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:14.146500 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:14.146511 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.146632 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.146955 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260812 11:15:14.148764 25283 heartbeater.cc:347] Connected to a master server at 127.23.97.254:46261
I20260812 11:15:14.149004 25283 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:14.149523 25283 heartbeater.cc:511] Master 127.23.97.254:46261 requested a full tablet report, sending...
I20260812 11:15:14.150547 24406 ts_manager.cc:194] Registered new tserver with Master: 28cc5378c8e04a26bf94e64c5e3951bf (127.23.97.195:34273)
I20260812 11:15:14.151121 24406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.195:52187
I20260812 11:15:14.155102 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:14.160507 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.157192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.23.97.254:46261)
0812 11:15:14.157309 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.157312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.157493 (+ 181us) server_negotiation.cc:407] Connection header received
0812 11:15:14.157590 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.157592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.157627 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.157686 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:14.157911 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.158382 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.159175 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.159376 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.159448 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.160134 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:14.160138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:14.160140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:14.160151 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:14.160166 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:14.160173 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:14.160238 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.160306 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.160347 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:15:14.163113 24406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:14.166221 24406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:14.180830 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.174300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33457 (local address 127.0.0.1:41062)
0812 11:15:14.175111 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.175165 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.175287 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.176246 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.176252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.176278 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.176419 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.176427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.177880 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.177884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.179240 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.179250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.180342 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.180459 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180481 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.180584 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.180668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":23,"thread_start_us":95,"threads_started":1}
I20260812 11:15:14.180881 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.175308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.23.97.193:33457)
0812 11:15:14.175724 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.175735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.175758 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:15:14.175829 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.175832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.176067 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.176203 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.176569 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.177744 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.179377 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.180120 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180216 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180412 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.180620 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.180725 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":156,"threads_started":1}
I20260812 11:15:14.180652 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.174259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35273 (local address 127.0.0.1:44144)
0812 11:15:14.175111 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.175164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.175290 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.176634 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.176637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.176654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.176747 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.176753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.177918 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.177924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.179037 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.179046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.180075 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.180191 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180233 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.180352 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.180504 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":66,"thread_start_us":91,"threads_started":1}
I20260812 11:15:14.181459 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.174654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.23.97.194:35273)
0812 11:15:14.176126 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.176138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.176187 (+ 49us) server_negotiation.cc:407] Connection header received
0812 11:15:14.176260 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.176264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.176471 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.176662 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.176840 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.177779 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.179167 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.179943 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180061 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180262 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.180752 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.180866 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":110,"threads_started":1}
I20260812 11:15:14.181550 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.175708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.23.97.195:34273)
0812 11:15:14.176025 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.176037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.176680 (+ 643us) server_negotiation.cc:407] Connection header received
0812 11:15:14.176755 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.176759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.176944 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.177089 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.177334 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.178406 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.179687 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.180352 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180478 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180641 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.181270 (+ 629us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.181368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
I20260812 11:15:14.181990 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.174632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:34273 (local address 127.0.0.1:44382)
0812 11:15:14.176458 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.176470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.176582 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.177101 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.177104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.177123 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.177206 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.177212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.178541 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.178544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.179572 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.179579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.180713 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.180716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.180805 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.180828 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.180901 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.181862 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260812 11:15:14.184188 25145 tablet_service.cc:1467] Processing CreateTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c (DEFAULT_TABLE table=test-table [id=ab4aa4d29ee44888b81d74ddd0edd63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.184187 24641 tablet_service.cc:1467] Processing CreateTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c (DEFAULT_TABLE table=test-table [id=ab4aa4d29ee44888b81d74ddd0edd63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.184355 24891 tablet_service.cc:1467] Processing CreateTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c (DEFAULT_TABLE table=test-table [id=ab4aa4d29ee44888b81d74ddd0edd63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:14.185308 24891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcac908824a64813bfd9bcf3fd54dd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.185318 24641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcac908824a64813bfd9bcf3fd54dd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.185318 25145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcac908824a64813bfd9bcf3fd54dd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:14.191478 25316 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Bootstrap starting.
I20260812 11:15:14.193024 25319 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Bootstrap starting.
I20260812 11:15:14.193902 25316 tablet_bootstrap.cc:654] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.194639 25316 log.cc:824] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.194815 25319 tablet_bootstrap.cc:654] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.195799 25319 log.cc:824] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.196547 25317 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Bootstrap starting.
I20260812 11:15:14.197043 25316 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: No bootstrap required, opened a new log
I20260812 11:15:14.197238 25316 ts_tablet_manager.cc:1397] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 11:15:14.198342 25317 tablet_bootstrap.cc:654] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:14.199020 25317 log.cc:824] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:14.200627 25319 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: No bootstrap required, opened a new log
I20260812 11:15:14.200839 25319 ts_tablet_manager.cc:1397] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 11:15:14.200843 25316 raft_consensus.cc:348] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.201134 25316 raft_consensus.cc:374] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.201220 25316 raft_consensus.cc:729] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cc5378c8e04a26bf94e64c5e3951bf, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.201593 25317 tablet_bootstrap.cc:492] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: No bootstrap required, opened a new log
I20260812 11:15:14.201767 25317 ts_tablet_manager.cc:1397] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260812 11:15:14.201733 25316 consensus_queue.cc:260] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.202736 25316 ts_tablet_manager.cc:1428] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260812 11:15:14.203117 25283 heartbeater.cc:503] Master 127.23.97.254:46261 was elected leader, sending a full tablet report...
I20260812 11:15:14.205129 25319 raft_consensus.cc:348] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.205366 25319 raft_consensus.cc:374] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.205437 25319 raft_consensus.cc:729] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aec3655515d46b8af3912d610d8345c, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.205405 25317 raft_consensus.cc:348] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.205658 25317 raft_consensus.cc:374] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:14.205730 25317 raft_consensus.cc:729] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78dc924742146beba8903f3b3edbf03, State: Initialized, Role: FOLLOWER
I20260812 11:15:14.205929 25319 consensus_queue.cc:260] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.206168 25317 consensus_queue.cc:260] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.206626 25018 heartbeater.cc:503] Master 127.23.97.254:46261 was elected leader, sending a full tablet report...
I20260812 11:15:14.206879 24772 heartbeater.cc:503] Master 127.23.97.254:46261 was elected leader, sending a full tablet report...
I20260812 11:15:14.207204 25317 ts_tablet_manager.cc:1428] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Time spent starting tablet: real 0.005s user 0.004s sys 0.003s
I20260812 11:15:14.207953 25319 ts_tablet_manager.cc:1428] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260812 11:15:14.257748 24773 tablet.cc:2369] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:14.346079 25331 raft_consensus.cc:482] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:14.346320 25331 raft_consensus.cc:504] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.347447 25331 leader_election.cc:288] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aec3655515d46b8af3912d610d8345c (127.23.97.194:35273), 28cc5378c8e04a26bf94e64c5e3951bf (127.23.97.195:34273)
I20260812 11:15:14.351392 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.347777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:46389 (local address 127.23.97.194:35273)
0812 11:15:14.348001 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.348007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.348097 (+ 90us) server_negotiation.cc:407] Connection header received
0812 11:15:14.348280 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.348283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.348337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.348459 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.348761 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.349278 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350366 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.350769 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350899 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.350952 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.351169 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.351250 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260812 11:15:14.351416 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.347842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:54415 (local address 127.23.97.195:34273)
0812 11:15:14.348120 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.348126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.348142 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:14.348334 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.348337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.348400 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.348491 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:14.348796 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.349291 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350386 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.350774 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350908 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.350964 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.351192 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.351266 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260812 11:15:14.351429 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.347684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:35273 (local address 127.23.97.193:46389)
0812 11:15:14.347991 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.348004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.348139 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.348477 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.348480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.348496 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.348635 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.348640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.349392 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.349398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.350234 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.350243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.350892 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.351041 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.351060 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.351151 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.351253 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":61,"spinlock_wait_cycles":14720,"thread_start_us":106,"threads_started":1}
I20260812 11:15:14.351440 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.347702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:34273 (local address 127.23.97.193:54415)
0812 11:15:14.348063 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.348077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:14.348177 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:14.348505 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:14.348508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:14.348550 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:14.348669 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.348674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.349416 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.349421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.350246 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:14.350255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.350901 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.350905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:14.351063 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.351081 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 11:15:14.351172 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:15:14.351262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":51,"spinlock_wait_cycles":4096,"thread_start_us":64,"threads_started":1}
I20260812 11:15:14.352172 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcac908824a64813bfd9bcf3fd54dd6c" candidate_uuid: "d78dc924742146beba8903f3b3edbf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" is_pre_election: true
I20260812 11:15:14.352172 24940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcac908824a64813bfd9bcf3fd54dd6c" candidate_uuid: "d78dc924742146beba8903f3b3edbf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aec3655515d46b8af3912d610d8345c" is_pre_election: true
I20260812 11:15:14.352494 25189 raft_consensus.cc:2454] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78dc924742146beba8903f3b3edbf03 in term 0.
I20260812 11:15:14.352515 24940 raft_consensus.cc:2454] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78dc924742146beba8903f3b3edbf03 in term 0.
I20260812 11:15:14.352962 24555 leader_election.cc:302] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cc5378c8e04a26bf94e64c5e3951bf, d78dc924742146beba8903f3b3edbf03; no voters:
I20260812 11:15:14.353235 25331 raft_consensus.cc:2781] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:14.353322 25331 raft_consensus.cc:482] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:14.353366 25331 raft_consensus.cc:3037] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.354379 25331 raft_consensus.cc:504] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.354700 25331 leader_election.cc:288] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [CANDIDATE]: Term 1 election: Requested vote from peers 5aec3655515d46b8af3912d610d8345c (127.23.97.194:35273), 28cc5378c8e04a26bf94e64c5e3951bf (127.23.97.195:34273)
I20260812 11:15:14.354995 24940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcac908824a64813bfd9bcf3fd54dd6c" candidate_uuid: "d78dc924742146beba8903f3b3edbf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aec3655515d46b8af3912d610d8345c"
I20260812 11:15:14.355058 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcac908824a64813bfd9bcf3fd54dd6c" candidate_uuid: "d78dc924742146beba8903f3b3edbf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cc5378c8e04a26bf94e64c5e3951bf"
I20260812 11:15:14.355123 24940 raft_consensus.cc:3037] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.355176 25189 raft_consensus.cc:3037] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:14.356262 24940 raft_consensus.cc:2454] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78dc924742146beba8903f3b3edbf03 in term 1.
I20260812 11:15:14.356297 25189 raft_consensus.cc:2454] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78dc924742146beba8903f3b3edbf03 in term 1.
I20260812 11:15:14.356640 24560 leader_election.cc:302] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aec3655515d46b8af3912d610d8345c, d78dc924742146beba8903f3b3edbf03; no voters:
I20260812 11:15:14.356818 25331 raft_consensus.cc:2781] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:14.357052 25331 raft_consensus.cc:686] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 1 LEADER]: Becoming Leader. State: Replica: d78dc924742146beba8903f3b3edbf03, State: Running, Role: LEADER
I20260812 11:15:14.357403 25331 consensus_queue.cc:237] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } }
I20260812 11:15:14.360737 24406 catalog_manager.cc:5495] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d78dc924742146beba8903f3b3edbf03 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "d78dc924742146beba8903f3b3edbf03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78dc924742146beba8903f3b3edbf03" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 } health_report { overall_health: UNKNOWN } } }
W20260812 11:15:14.388234 25286 tablet.cc:2369] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:14.413771 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.410081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.23.97.193:33457)
0812 11:15:14.410216 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:14.410220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:14.410260 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:15:14.410338 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:14.410340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:14.410401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:14.410483 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:14.410739 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.411237 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.412105 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:14.412326 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:14.412493 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:14.412804 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:14.413361 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:14.413453 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:14.413530 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 11:15:14.413586 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:15:14.419862 24940 raft_consensus.cc:1261] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 1 FOLLOWER]: Refusing update from remote peer d78dc924742146beba8903f3b3edbf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:14.419867 25189 raft_consensus.cc:1261] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 1 FOLLOWER]: Refusing update from remote peer d78dc924742146beba8903f3b3edbf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:14.420584 25335 consensus_queue.cc:1035] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aec3655515d46b8af3912d610d8345c" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 35273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:14.420791 25331 consensus_queue.cc:1035] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28cc5378c8e04a26bf94e64c5e3951bf" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:14.427825 25340 mvcc.cc:205] Tried to move back new op lower bound from 7317640455856680960 to 7317640455610449920. Current Snapshot: MvccSnapshot[applied={T|T < 7317640455856680960}]
I20260812 11:15:14.437580 24402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:14.437768 24402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:14.440222 24402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c07a3b732dba4ca092eb52ffa5539904: Sending DeleteTablet for 3 replicas of tablet fcac908824a64813bfd9bcf3fd54dd6c
I20260812 11:15:14.440990 24641 tablet_service.cc:1514] Processing DeleteTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='slave'} at 127.0.0.1:41062
I20260812 11:15:14.440994 25145 tablet_service.cc:1514] Processing DeleteTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='slave'} at 127.0.0.1:44382
I20260812 11:15:14.441498 25356 tablet_replica.cc:330] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03: stopping tablet replica
I20260812 11:15:14.441531 24891 tablet_service.cc:1514] Processing DeleteTablet for tablet fcac908824a64813bfd9bcf3fd54dd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:14 UTC) from {username='slave'} at 127.0.0.1:44144
I20260812 11:15:14.441828 25357 tablet_replica.cc:330] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: stopping tablet replica
I20260812 11:15:14.442015 25358 tablet_replica.cc:330] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: stopping tablet replica
I20260812 11:15:14.442340 25357 raft_consensus.cc:2229] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:14.442344 25358 raft_consensus.cc:2229] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:14.442395 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24523
I20260812 11:15:14.442600 25356 raft_consensus.cc:2229] T fcac908824a64813bfd9bcf3fd54dd6c P d78dc924742146beba8903f3b3edbf03 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:14.442941 25357 raft_consensus.cc:2258] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:14.446694 25358 raft_consensus.cc:2258] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:14.447427 25357 ts_tablet_manager.cc:1905] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:14.450501 25358 ts_tablet_manager.cc:1905] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:14.450824 25357 ts_tablet_manager.cc:1918] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:14.450968 25357 log.cc:1196] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/fcac908824a64813bfd9bcf3fd54dd6c
I20260812 11:15:14.451325 25357 ts_tablet_manager.cc:1939] T fcac908824a64813bfd9bcf3fd54dd6c P 28cc5378c8e04a26bf94e64c5e3951bf: Deleting consensus metadata
I20260812 11:15:14.452498 24368 catalog_manager.cc:4841] TS 28cc5378c8e04a26bf94e64c5e3951bf (127.23.97.195:34273): tablet fcac908824a64813bfd9bcf3fd54dd6c (table test-table [id=ab4aa4d29ee44888b81d74ddd0edd63c]) successfully deleted
I20260812 11:15:14.454087 25358 ts_tablet_manager.cc:1918] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:14.454203 25358 log.cc:1196] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/fcac908824a64813bfd9bcf3fd54dd6c
I20260812 11:15:14.454542 25358 ts_tablet_manager.cc:1939] T fcac908824a64813bfd9bcf3fd54dd6c P 5aec3655515d46b8af3912d610d8345c: Deleting consensus metadata
I20260812 11:15:14.455781 24375 catalog_manager.cc:4841] TS 5aec3655515d46b8af3912d610d8345c (127.23.97.194:35273): tablet fcac908824a64813bfd9bcf3fd54dd6c (table test-table [id=ab4aa4d29ee44888b81d74ddd0edd63c]) successfully deleted
W20260812 11:15:14.458290 24374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:33457 (error 108)
I20260812 11:15:14.458609 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24782
I20260812 11:15:14.461793 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:14.461412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:33457 (local address 127.0.0.1:41080)
0812 11:15:14.461593 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:14.461683 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:33457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 11:15:14.462237 24374 catalog_manager.cc:4584] TS d78dc924742146beba8903f3b3edbf03 (127.23.97.193:33457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcac908824a64813bfd9bcf3fd54dd6c: Network error: Client connection negotiation failed: client connection to 127.23.97.193:33457: connect: Connection refused (error 111)
I20260812 11:15:14.469703 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25022
I20260812 11:15:14.495770 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 24330
2026-08-12T11:15:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:14 dist-test-slave-zpvx krb5kdc[25370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:14 dist-test-slave-zpvx krb5kdc[25370](info): set up 2 sockets
Aug 12 11:15:14 dist-test-slave-zpvx krb5kdc[25370](info): commencing operation
krb5kdc: starting...
W20260812 11:15:15.817866 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:15 dist-test-slave-zpvx krb5kdc[25370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:15.944928 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35493
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:35493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:16.058034 25390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.058328 25390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.058401 25390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.062436 25390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:16.062516 25390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.062556 25390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.062615 25390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:16.062655 25390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:16.067692 25390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:35493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:16.068709 25390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.069712 25390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:16.078253 25409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.078243 25405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.078243 25404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.078270 25390 server_base.cc:1034] running on GCE node
I20260812 11:15:16.078986 25390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.080022 25390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.081220 25390 hybrid_clock.cc:648] HybridClock initialized: now 1786533316081196 us; error 43 us; skew 500 ppm
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:16.083951 25390 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:16.084878 25390 webserver.cc:460] Webserver started at http://127.23.97.254:35629/ using document root <none> and password file <none>
I20260812 11:15:16.085426 25390 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.085517 25390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.085752 25390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.087744 25390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "5d9c2f93da084240922d890da257cd3d"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6fa8649b69b94e805f862c979a6dc729"
server_key_iv: "d3b3a64b208a82ee5a655cac3c170237"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.088320 25390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "5d9c2f93da084240922d890da257cd3d"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6fa8649b69b94e805f862c979a6dc729"
server_key_iv: "d3b3a64b208a82ee5a655cac3c170237"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.091900 25390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:16.094225 25420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.095348 25390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:15:16.095500 25390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "5d9c2f93da084240922d890da257cd3d"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6fa8649b69b94e805f862c979a6dc729"
server_key_iv: "d3b3a64b208a82ee5a655cac3c170237"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.095610 25390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.125063 25390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.127869 25390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.128074 25390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.135066 25390 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:35493
I20260812 11:15:16.135093 25517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:35493 every 8 connection(s)
I20260812 11:15:16.135994 25390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:16.138665 25519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.141768 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25390
I20260812 11:15:16.141942 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:16.142328 23943 external_mini_cluster.cc:1428] Setting key 45824eb1439364aa75ac06bdb047ed03
I20260812 11:15:16.144873 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: Bootstrap starting.
I20260812 11:15:16.147025 25519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.147749 25519 log.cc:824] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.149602 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: No bootstrap required, opened a new log
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:16.153419 25519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } }
I20260812 11:15:16.153643 25519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.153714 25519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9c2f93da084240922d890da257cd3d, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.154179 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } }
I20260812 11:15:16.154337 25519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:16.154425 25519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:16.154551 25519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:16.155416 25519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } }
I20260812 11:15:16.155766 25519 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d9c2f93da084240922d890da257cd3d; no voters:
I20260812 11:15:16.156128 25519 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:16.156327 25527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:16.156606 25527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [term 1 LEADER]: Becoming Leader. State: Replica: 5d9c2f93da084240922d890da257cd3d, State: Running, Role: LEADER
I20260812 11:15:16.157080 25527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } }
I20260812 11:15:16.157407 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.143794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.23.97.254:35493)
0812 11:15:16.144282 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.144297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.144333 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:15:16.145055 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.145074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.145402 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.145731 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.146578 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.147612 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.148601 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.149060 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.149261 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.151981 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.151986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.152000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:16.152028 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.154012 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.154505 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.154529 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.154537 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.154702 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.155007 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.155011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.155012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.155403 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.155622 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.155929 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.156279 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260812 11:15:16.157466 25519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:16.159732 25534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9c2f93da084240922d890da257cd3d. Latest consensus state: current_term: 1 leader_uuid: "5d9c2f93da084240922d890da257cd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } } }
I20260812 11:15:16.159991 25534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.159955 25530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d9c2f93da084240922d890da257cd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d9c2f93da084240922d890da257cd3d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 35493 } } }
I20260812 11:15:16.160054 25530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:16.160354 25551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:16.163087 25551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:16.168256 25551 catalog_manager.cc:1347] Generated new cluster ID: 0febb643ca4145abb7796ee1663a9b5c
I20260812 11:15:16.168321 25551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:16.174762 25551 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:16.175967 25551 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:16.183447 25551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: Generated new TSK 0
I20260812 11:15:16.184139 25551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:15:16.222915 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:16.335881 25567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.336166 25567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.336256 25567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.340209 25567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.340298 25567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.340409 25567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:16.345592 25567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:16.346655 25567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.347695 25567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:16.357771 25567 server_base.cc:1034] running on GCE node
W20260812 11:15:16.357614 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.357542 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.357542 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.358366 25567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.359020 25567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.360349 25567 hybrid_clock.cc:648] HybridClock initialized: now 1786533316360316 us; error 43 us; skew 500 ppm
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:16.363361 25567 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:15:16.364228 25567 webserver.cc:460] Webserver started at http://127.23.97.193:34053/ using document root <none> and password file <none>
I20260812 11:15:16.364852 25567 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.364946 25567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.365190 25567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.367161 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "3324da27d2094c27bac9ec29e5f77d38"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "f9ff2f5cc413622bdbffa60855ae9e15"
server_key_iv: "5463e6d3358eaaacae78fd41e1925aa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.367733 25567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "3324da27d2094c27bac9ec29e5f77d38"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "f9ff2f5cc413622bdbffa60855ae9e15"
server_key_iv: "5463e6d3358eaaacae78fd41e1925aa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.371417 25567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:15:16.373917 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.375080 25567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:16.375226 25567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "3324da27d2094c27bac9ec29e5f77d38"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "f9ff2f5cc413622bdbffa60855ae9e15"
server_key_iv: "5463e6d3358eaaacae78fd41e1925aa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.375334 25567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.393930 25567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.397265 25567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.397482 25567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.398083 25567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:16.398996 25567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:16.399070 25567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.399168 25567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:16.399219 25567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.408895 25567 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:44897
I20260812 11:15:16.408977 25828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:44897 every 8 connection(s)
I20260812 11:15:16.409828 25567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:16.419714 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25567
I20260812 11:15:16.419909 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:16.420264 23943 external_mini_cluster.cc:1428] Setting key d3d50576ee394801f1d58c227f84b43f
I20260812 11:15:16.423507 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.411679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:56235 (local address 127.23.97.254:35493)
0812 11:15:16.411825 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.411829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.412683 (+ 854us) server_negotiation.cc:407] Connection header received
0812 11:15:16.413972 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.413975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.414021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.414095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.415225 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.415755 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.416908 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.417146 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.417230 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.419595 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.419598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.419600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:16.419625 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.421256 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.421873 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.421876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.421878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.421973 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.422315 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.422321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.422326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.422572 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.422677 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.423119 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.423269 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:16.424182 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.412034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35493 (local address 127.23.97.193:56235)
0812 11:15:16.412530 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.412572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.413779 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.414254 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.414260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.414633 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:16.415092 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.415105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.415862 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.415865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.416792 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.416802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.417290 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.417293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.417429 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.417946 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:16.417961 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:16.419386 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:16.421366 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.421372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.421386 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.421760 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.422068 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.422071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.422072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.422201 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.422676 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:16.422681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:16.422891 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.423040 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.423271 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260812 11:15:16.425453 25829 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35493
I20260812 11:15:16.425674 25829 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:16.426129 25829 heartbeater.cc:511] Master 127.23.97.254:35493 requested a full tablet report, sending...
I20260812 11:15:16.427544 25445 ts_manager.cc:194] Registered new tserver with Master: 3324da27d2094c27bac9ec29e5f77d38 (127.23.97.193:44897)
I20260812 11:15:16.428702 25445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:56235
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:15:16.448843 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:16.560247 25837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.560493 25837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.560571 25837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.564546 25837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.564600 25837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.564700 25837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:16.569800 25837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:16.570778 25837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.571766 25837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:16.581413 25860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.581413 25849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.581444 25848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.581650 25837 server_base.cc:1034] running on GCE node
I20260812 11:15:16.582068 25837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.582692 25837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.583884 25837 hybrid_clock.cc:648] HybridClock initialized: now 1786533316583861 us; error 34 us; skew 500 ppm
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:16.586706 25837 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:15:16.587530 25837 webserver.cc:460] Webserver started at http://127.23.97.194:44163/ using document root <none> and password file <none>
I20260812 11:15:16.588084 25837 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.588174 25837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.588466 25837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.590494 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "9bc2740748cc42af9188c3fba9b44e74"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6a36feed58457c72fe8145142f3e346f"
server_key_iv: "8a6130b775d41e3315c7b007535fc1d4"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.591050 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "9bc2740748cc42af9188c3fba9b44e74"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6a36feed58457c72fe8145142f3e346f"
server_key_iv: "8a6130b775d41e3315c7b007535fc1d4"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.594677 25837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 11:15:16.596954 25873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.598119 25837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:16.598253 25837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "9bc2740748cc42af9188c3fba9b44e74"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "6a36feed58457c72fe8145142f3e346f"
server_key_iv: "8a6130b775d41e3315c7b007535fc1d4"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.598371 25837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.623100 25837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.626102 25837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.626322 25837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.626910 25837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:16.627748 25837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:16.627820 25837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.627894 25837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:16.627938 25837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.636962 25837 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:45651
I20260812 11:15:16.636991 26068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:45651 every 8 connection(s)
I20260812 11:15:16.637862 25837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:16.645215 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 25837
I20260812 11:15:16.645371 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:16.645687 23943 external_mini_cluster.cc:1428] Setting key 401cd4c7726f5658d4ab6f3e05141e45
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:16.652578 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.640437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:53789 (local address 127.23.97.254:35493)
0812 11:15:16.640575 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.640579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.640895 (+ 316us) server_negotiation.cc:407] Connection header received
0812 11:15:16.642698 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.642701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.642751 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.642832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.643893 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.644543 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.646055 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.646290 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.646378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.648943 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.648947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.648949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:16.648973 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.650451 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.650996 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.651005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.651006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.651084 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.651450 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.651454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.651456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.651648 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.651759 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.652249 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.652403 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 11:15:16.653375 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.640220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35493 (local address 127.23.97.194:53789)
0812 11:15:16.640737 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.640789 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.642445 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.642959 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.642966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.643294 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:16.643738 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.643750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.644657 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.644660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.645929 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.645942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.646441 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.646446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.646613 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.647248 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:16.647264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:16.648766 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:16.650581 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.650587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.650602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.650877 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.651189 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.651192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.651194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.651348 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.651759 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:16.651767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:16.652011 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.652203 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.652547 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":164,"threads_started":1}
I20260812 11:15:16.654666 26069 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35493
I20260812 11:15:16.654910 26069 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:16.655404 26069 heartbeater.cc:511] Master 127.23.97.254:35493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:15:16.656673 25445 ts_manager.cc:194] Registered new tserver with Master: 9bc2740748cc42af9188c3fba9b44e74 (127.23.97.194:45651)
I20260812 11:15:16.657270 25445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:53789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:15:16.676208 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:16.789712 26080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:16.789951 26080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:16.790025 26080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:16.793967 26080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:16.794021 26080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:16.794123 26080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:16.799296 26080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:16.800292 26080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:16.801308 26080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:16.810936 26089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.810943 26092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:16.810936 26098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:16.810989 26080 server_base.cc:1034] running on GCE node
I20260812 11:15:16.811551 26080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:16.812160 26080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:16.813323 26080 hybrid_clock.cc:648] HybridClock initialized: now 1786533316813308 us; error 57 us; skew 500 ppm
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:16.815980 26080 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:15:16.816815 26080 webserver.cc:460] Webserver started at http://127.23.97.195:36467/ using document root <none> and password file <none>
I20260812 11:15:16.817354 26080 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:16.817416 26080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:16.817581 26080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:16.819572 26080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "8ae27da0f7f0417cae8928d8a3b385a4"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "729d26b2e911c777d3436e7721ab74fc"
server_key_iv: "cee7b74e5c2b52a00bc51c90a8884803"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.820128 26080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae27da0f7f0417cae8928d8a3b385a4"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "729d26b2e911c777d3436e7721ab74fc"
server_key_iv: "cee7b74e5c2b52a00bc51c90a8884803"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.824301 26080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 11:15:16.826713 26111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.827921 26080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:16.828017 26080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "8ae27da0f7f0417cae8928d8a3b385a4"
format_stamp: "Formatted at 2026-08-12 11:15:16 on dist-test-slave-zpvx"
server_key: "729d26b2e911c777d3436e7721ab74fc"
server_key_iv: "cee7b74e5c2b52a00bc51c90a8884803"
server_key_version: "encryptionkey@0"
I20260812 11:15:16.828094 26080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:16.843581 26080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:16.846767 26080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:16.846917 26080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:16.847421 26080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:16.848167 26080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:16.848214 26080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.848261 26080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:16.848282 26080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:16.857257 26080 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:36335
I20260812 11:15:16.857334 26300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:36335 every 8 connection(s)
I20260812 11:15:16.858301 26080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:16.862862 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26080
I20260812 11:15:16.863039 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:16.863409 23943 external_mini_cluster.cc:1428] Setting key 58b70c98c33bed5df969445d0b815ed6
Aug 12 11:15:16 dist-test-slave-zpvx krb5kdc[25370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:16.872265 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.860273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:42439 (local address 127.23.97.254:35493)
0812 11:15:16.860466 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.860470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.861315 (+ 845us) server_negotiation.cc:407] Connection header received
0812 11:15:16.862770 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.862773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.862821 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.862897 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.864487 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.865108 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.866391 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.866616 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.866733 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.869208 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.869211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.869213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:16.869237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.870483 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.870959 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.870961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.870962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.871033 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.871355 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.871357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.871358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.871524 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.871599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.872046 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.872146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:15:16.873133 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.860677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:35493 (local address 127.23.97.195:42439)
0812 11:15:16.861138 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.861209 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.862511 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.863121 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.863130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.863648 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:16.864300 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.864324 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.865242 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.865245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.866259 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.866270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.866777 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.866782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.866949 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.867506 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:16.867521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:16.869030 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:16.870588 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.870593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.870604 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.870859 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.871108 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:16.871111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:16.871112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:16.871264 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:16.871609 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:16.871614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:16.871841 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.871986 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.872251 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":130,"threads_started":1}
I20260812 11:15:16.874259 26302 heartbeater.cc:347] Connected to a master server at 127.23.97.254:35493
I20260812 11:15:16.874513 26302 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:16.875021 26302 heartbeater.cc:511] Master 127.23.97.254:35493 requested a full tablet report, sending...
I20260812 11:15:16.876127 25445 ts_manager.cc:194] Registered new tserver with Master: 8ae27da0f7f0417cae8928d8a3b385a4 (127.23.97.195:36335)
I20260812 11:15:16.876838 25445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:42439
I20260812 11:15:16.877588 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:16.886703 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.879604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.23.97.254:35493)
0812 11:15:16.879710 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.879732 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.879957 (+ 225us) server_negotiation.cc:407] Connection header received
0812 11:15:16.880067 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.880069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.880103 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.880167 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:16.880871 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.881358 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.882146 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.882350 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.882424 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.883903 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:16.883906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:16.883908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:16.883928 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:16.885193 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.885638 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.885640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.885642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.885712 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:16.886020 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:16.886023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:16.886024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:16.886196 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:16.886319 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.886522 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.886595 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:15:16.889479 25445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:16.892481 25445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:16.907272 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.901252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:36335 (local address 127.0.0.1:56970)
0812 11:15:16.902003 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.902056 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.902196 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.903156 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.903162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.903186 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:16.903368 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.903378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.904531 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.904534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.905787 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.905797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.906719 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.906722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.906838 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.906878 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.906990 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.907104 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260812 11:15:16.907289 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.901390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.23.97.195:36335)
0812 11:15:16.902512 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.902531 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.902614 (+ 83us) server_negotiation.cc:407] Connection header received
0812 11:15:16.902750 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.902756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.902980 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.903140 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:16.903513 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.904327 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.905910 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.906589 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.906699 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.906854 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.907036 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.907140 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":113,"threads_started":1}
I20260812 11:15:16.908144 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.901343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:45651 (local address 127.0.0.1:57840)
0812 11:15:16.902604 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.902618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.902714 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.903304 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.903307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.903323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:16.903514 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.903522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.905026 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.905030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.906309 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.906321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.907643 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.907647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.907802 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.907827 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.907907 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.907979 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":131,"thread_start_us":136,"threads_started":1}
I20260812 11:15:16.908234 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.901865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.23.97.194:45651)
0812 11:15:16.902327 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.902340 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.902690 (+ 350us) server_negotiation.cc:407] Connection header received
0812 11:15:16.902833 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.902837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.903031 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.903191 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:16.903650 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.904856 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.906464 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.907321 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.907435 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.907576 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.907973 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.908079 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":124,"threads_started":1}
I20260812 11:15:16.908541 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.902368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.23.97.193:44897)
0812 11:15:16.903093 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:16.903112 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:16.903133 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:15:16.903204 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:16.903209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:16.903401 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:16.903543 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:16.903709 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.904891 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.906502 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.907330 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.907451 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.907591 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:16.908274 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 11:15:16.908403 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":121,"threads_started":1}
I20260812 11:15:16.908170 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:16.901945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44897 (local address 127.0.0.1:51430)
0812 11:15:16.902756 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:16.902770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:16.902851 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:16.903531 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:16.903534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:16.903552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:16.903628 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.903633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.905026 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.905030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.906360 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:16.906372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:16.907652 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:16.907656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:16.907818 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:16.907837 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:15:16.907928 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:15:16.908004 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":204,"thread_start_us":94,"threads_started":1}
I20260812 11:15:16.911361 25952 tablet_service.cc:1467] Processing CreateTablet for tablet 20fc249eafd54a19aed254767b48e23d (DEFAULT_TABLE table=test-table [id=9b5321feaf774a58bbaf6cad32bb1ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:16.911399 26189 tablet_service.cc:1467] Processing CreateTablet for tablet 20fc249eafd54a19aed254767b48e23d (DEFAULT_TABLE table=test-table [id=9b5321feaf774a58bbaf6cad32bb1ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:16.911399 25692 tablet_service.cc:1467] Processing CreateTablet for tablet 20fc249eafd54a19aed254767b48e23d (DEFAULT_TABLE table=test-table [id=9b5321feaf774a58bbaf6cad32bb1ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:16.912349 25952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fc249eafd54a19aed254767b48e23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.912441 26189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fc249eafd54a19aed254767b48e23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.912441 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fc249eafd54a19aed254767b48e23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:16.918377 26330 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Bootstrap starting.
I20260812 11:15:16.919443 26331 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Bootstrap starting.
I20260812 11:15:16.920264 26334 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: Bootstrap starting.
I20260812 11:15:16.921489 26330 tablet_bootstrap.cc:654] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.921478 26331 tablet_bootstrap.cc:654] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.922400 26334 tablet_bootstrap.cc:654] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:16.922959 26331 log.cc:824] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.922959 26330 log.cc:824] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.923610 26334 log.cc:824] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:16.925158 26330 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: No bootstrap required, opened a new log
I20260812 11:15:16.925160 26331 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: No bootstrap required, opened a new log
I20260812 11:15:16.925365 26331 ts_tablet_manager.cc:1397] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 11:15:16.925374 26330 ts_tablet_manager.cc:1397] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 11:15:16.925505 26334 tablet_bootstrap.cc:492] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: No bootstrap required, opened a new log
I20260812 11:15:16.925691 26334 ts_tablet_manager.cc:1397] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:16.928813 26334 raft_consensus.cc:348] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.929059 26334 raft_consensus.cc:374] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.929147 26334 raft_consensus.cc:729] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3324da27d2094c27bac9ec29e5f77d38, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.929334 26331 raft_consensus.cc:348] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.929334 26330 raft_consensus.cc:348] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.929581 26330 raft_consensus.cc:374] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.929581 26331 raft_consensus.cc:374] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:16.929642 26331 raft_consensus.cc:729] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae27da0f7f0417cae8928d8a3b385a4, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.929713 26330 raft_consensus.cc:729] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc2740748cc42af9188c3fba9b44e74, State: Initialized, Role: FOLLOWER
I20260812 11:15:16.929658 26334 consensus_queue.cc:260] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.930055 26331 consensus_queue.cc:260] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.930088 26330 consensus_queue.cc:260] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:16.930275 25829 heartbeater.cc:503] Master 127.23.97.254:35493 was elected leader, sending a full tablet report...
I20260812 11:15:16.930675 26334 ts_tablet_manager.cc:1428] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260812 11:15:16.931027 26331 ts_tablet_manager.cc:1428] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:16.931027 26330 ts_tablet_manager.cc:1428] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:16.931111 26302 heartbeater.cc:503] Master 127.23.97.254:35493 was elected leader, sending a full tablet report...
I20260812 11:15:16.931336 26069 heartbeater.cc:503] Master 127.23.97.254:35493 was elected leader, sending a full tablet report...
I20260812 11:15:17.021775 26341 raft_consensus.cc:482] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:17.022025 26341 raft_consensus.cc:504] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:17.023245 26341 leader_election.cc:288] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae27da0f7f0417cae8928d8a3b385a4 (127.23.97.195:36335), 3324da27d2094c27bac9ec29e5f77d38 (127.23.97.193:44897)
I20260812 11:15:17.028146 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.023667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:49249 (local address 127.23.97.193:44897)
0812 11:15:17.023815 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.023820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.024068 (+ 248us) server_negotiation.cc:407] Connection header received
0812 11:15:17.024170 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.024174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.024226 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.024312 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:17.024706 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.025364 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.026772 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.027401 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.027529 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.027596 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.027935 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.028018 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:15:17.028146 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.023616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:43705 (local address 127.23.97.195:36335)
0812 11:15:17.023752 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.023757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.023774 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:15:17.023959 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.023963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.024037 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.024121 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:17.024505 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.025206 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.026772 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.027393 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.027529 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.027584 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.027935 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.028018 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:15:17.028146 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.023501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:36335 (local address 127.23.97.194:43705)
0812 11:15:17.023700 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.023717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.023822 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.024147 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.024150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.024170 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:17.024343 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.024351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.025342 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.025346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.026602 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.026613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.027530 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.027534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.027711 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.027789 (+ 78us) client_negotiation.cc:769] Sending connection context
0812 11:15:17.027873 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:15:17.027989 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":82}
I20260812 11:15:17.028154 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.023529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44897 (local address 127.23.97.194:49249)
0812 11:15:17.023962 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.023975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.024082 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.024334 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.024337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.024403 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:17.024525 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.024532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.025486 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.025489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.026629 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.026637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.027530 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.027534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.027741 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.027800 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 11:15:17.027873 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:15:17.027991 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":120,"thread_start_us":138,"threads_started":1}
I20260812 11:15:17.028817 25734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fc249eafd54a19aed254767b48e23d" candidate_uuid: "9bc2740748cc42af9188c3fba9b44e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3324da27d2094c27bac9ec29e5f77d38" is_pre_election: true
I20260812 11:15:17.028952 26219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fc249eafd54a19aed254767b48e23d" candidate_uuid: "9bc2740748cc42af9188c3fba9b44e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" is_pre_election: true
I20260812 11:15:17.029148 25734 raft_consensus.cc:2454] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc2740748cc42af9188c3fba9b44e74 in term 0.
I20260812 11:15:17.029299 26219 raft_consensus.cc:2454] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc2740748cc42af9188c3fba9b44e74 in term 0.
I20260812 11:15:17.029685 25877 leader_election.cc:302] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3324da27d2094c27bac9ec29e5f77d38, 9bc2740748cc42af9188c3fba9b44e74; no voters:
I20260812 11:15:17.029959 26341 raft_consensus.cc:2781] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:17.030036 26341 raft_consensus.cc:482] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:17.030066 26341 raft_consensus.cc:3037] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.031211 26341 raft_consensus.cc:504] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:17.031575 26341 leader_election.cc:288] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [CANDIDATE]: Term 1 election: Requested vote from peers 8ae27da0f7f0417cae8928d8a3b385a4 (127.23.97.195:36335), 3324da27d2094c27bac9ec29e5f77d38 (127.23.97.193:44897)
I20260812 11:15:17.031937 25734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fc249eafd54a19aed254767b48e23d" candidate_uuid: "9bc2740748cc42af9188c3fba9b44e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3324da27d2094c27bac9ec29e5f77d38"
I20260812 11:15:17.031924 26219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fc249eafd54a19aed254767b48e23d" candidate_uuid: "9bc2740748cc42af9188c3fba9b44e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae27da0f7f0417cae8928d8a3b385a4"
I20260812 11:15:17.032048 25734 raft_consensus.cc:3037] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.032063 26219 raft_consensus.cc:3037] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.033385 26219 raft_consensus.cc:2454] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc2740748cc42af9188c3fba9b44e74 in term 1.
I20260812 11:15:17.033385 25734 raft_consensus.cc:2454] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc2740748cc42af9188c3fba9b44e74 in term 1.
I20260812 11:15:17.033691 25878 leader_election.cc:302] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae27da0f7f0417cae8928d8a3b385a4, 9bc2740748cc42af9188c3fba9b44e74; no voters:
I20260812 11:15:17.033862 26341 raft_consensus.cc:2781] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:17.034044 26341 raft_consensus.cc:686] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc2740748cc42af9188c3fba9b44e74, State: Running, Role: LEADER
I20260812 11:15:17.034371 26341 consensus_queue.cc:237] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } }
I20260812 11:15:17.037628 25442 catalog_manager.cc:5495] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc2740748cc42af9188c3fba9b44e74 (127.23.97.194). New cstate: current_term: 1 leader_uuid: "9bc2740748cc42af9188c3fba9b44e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc2740748cc42af9188c3fba9b44e74" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 45651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:17.063336 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.059873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.23.97.194:45651)
0812 11:15:17.060007 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.060011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.060023 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:17.060141 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.060144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.060191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.060288 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:17.060520 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.061010 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.061884 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.062095 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.062176 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.062511 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:17.062897 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:17.063004 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.063094 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.063152 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:15:17.069010 25734 raft_consensus.cc:1261] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38 [term 1 FOLLOWER]: Refusing update from remote peer 9bc2740748cc42af9188c3fba9b44e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:17.069008 26219 raft_consensus.cc:1261] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 1 FOLLOWER]: Refusing update from remote peer 9bc2740748cc42af9188c3fba9b44e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:17.069759 26341 consensus_queue.cc:1035] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae27da0f7f0417cae8928d8a3b385a4" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 36335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:17.069995 26348 consensus_queue.cc:1035] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3324da27d2094c27bac9ec29e5f77d38" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:17.078637 26353 mvcc.cc:205] Tried to move back new op lower bound from 7317640466707488768 to 7317640466575257600. Current Snapshot: MvccSnapshot[applied={T|T < 7317640466707488768}]
I20260812 11:15:17.079905 26359 mvcc.cc:205] Tried to move back new op lower bound from 7317640466707488768 to 7317640466575257600. Current Snapshot: MvccSnapshot[applied={T|T < 7317640466707488768}]
I20260812 11:15:17.086521 25442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:17.086665 25442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:17.089128 25442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d9c2f93da084240922d890da257cd3d: Sending DeleteTablet for 3 replicas of tablet 20fc249eafd54a19aed254767b48e23d
I20260812 11:15:17.089920 25692 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fc249eafd54a19aed254767b48e23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:17 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:51430
I20260812 11:15:17.089912 26189 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fc249eafd54a19aed254767b48e23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:17 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:56970
I20260812 11:15:17.090132 25952 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fc249eafd54a19aed254767b48e23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:17 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:57840
I20260812 11:15:17.090406 26376 tablet_replica.cc:330] T 20fc249eafd54a19aed254767b48e23d P 3324da27d2094c27bac9ec29e5f77d38: stopping tablet replica
I20260812 11:15:17.090721 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25567
I20260812 11:15:17.123876 26375 tablet_replica.cc:330] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: stopping tablet replica
I20260812 11:15:17.124336 26375 raft_consensus.cc:2229] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:17.124773 26375 raft_consensus.cc:2258] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:17.124886 26378 tablet_replica.cc:330] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: stopping tablet replica
I20260812 11:15:17.125182 26378 raft_consensus.cc:2229] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:17.125578 26378 raft_consensus.cc:2258] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:17.127305 26378 ts_tablet_manager.cc:1905] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:17.127311 26375 ts_tablet_manager.cc:1905] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:17.130136 26378 ts_tablet_manager.cc:1918] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:17.130136 26375 ts_tablet_manager.cc:1918] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:17.130239 26375 log.cc:1196] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/20fc249eafd54a19aed254767b48e23d
I20260812 11:15:17.130323 26378 log.cc:1196] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/20fc249eafd54a19aed254767b48e23d
I20260812 11:15:17.130676 26375 ts_tablet_manager.cc:1939] T 20fc249eafd54a19aed254767b48e23d P 8ae27da0f7f0417cae8928d8a3b385a4: Deleting consensus metadata
I20260812 11:15:17.130697 26378 ts_tablet_manager.cc:1939] T 20fc249eafd54a19aed254767b48e23d P 9bc2740748cc42af9188c3fba9b44e74: Deleting consensus metadata
I20260812 11:15:17.132104 25427 catalog_manager.cc:4841] TS 8ae27da0f7f0417cae8928d8a3b385a4 (127.23.97.195:36335): tablet 20fc249eafd54a19aed254767b48e23d (table test-table [id=9b5321feaf774a58bbaf6cad32bb1ffe]) successfully deleted
I20260812 11:15:17.132112 25424 catalog_manager.cc:4841] TS 9bc2740748cc42af9188c3fba9b44e74 (127.23.97.194:45651): tablet 20fc249eafd54a19aed254767b48e23d (table test-table [id=9b5321feaf774a58bbaf6cad32bb1ffe]) successfully deleted
W20260812 11:15:17.132669 25424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:44897 (error 108)
I20260812 11:15:17.132967 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25837
I20260812 11:15:17.146891 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.146517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:44897 (local address 127.0.0.1:51436)
0812 11:15:17.146723 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.146793 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260812 11:15:17.147341 25424 catalog_manager.cc:4584] TS 3324da27d2094c27bac9ec29e5f77d38 (127.23.97.193:44897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20fc249eafd54a19aed254767b48e23d: Network error: Client connection negotiation failed: client connection to 127.23.97.193:44897: connect: Connection refused (error 111)
I20260812 11:15:17.161442 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26080
I20260812 11:15:17.175207 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 25390
2026-08-12T11:15:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2687 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T11:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:17Z Disabled control of system clock
I20260812 11:15:17.222115 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34131
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:17.345865 26389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.346271 26389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.346338 26389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.354120 26389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:17.354218 26389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.354249 26389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.354277 26389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:17.354350 26389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:17.362362 26389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:17.363549 26389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.365087 26389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:17.375383 26404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.375412 26406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.375820 26408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.375790 26389 server_base.cc:1034] running on GCE node
I20260812 11:15:17.376605 26389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:17.377800 26389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:17.379001 26389 hybrid_clock.cc:648] HybridClock initialized: now 1786533317378974 us; error 50 us; skew 500 ppm
I20260812 11:15:17.380905 26389 webserver.cc:460] Webserver started at http://127.23.97.254:36931/ using document root <none> and password file <none>
I20260812 11:15:17.381495 26389 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:17.381590 26389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:17.381824 26389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:17.383910 26389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "708e6d5de4da49d5887a19959a95658f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "25d94a2580685fc2c99d9b138271d437"
server_key_iv: "956c47bb8d23921057a3435072c0cbb2"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.384562 26389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "708e6d5de4da49d5887a19959a95658f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "25d94a2580685fc2c99d9b138271d437"
server_key_iv: "956c47bb8d23921057a3435072c0cbb2"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.388149 26389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:17.390625 26415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.391772 26389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:17.391914 26389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "708e6d5de4da49d5887a19959a95658f"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "25d94a2580685fc2c99d9b138271d437"
server_key_iv: "956c47bb8d23921057a3435072c0cbb2"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.392024 26389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:17.403817 26389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:17.404544 26389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:17.404738 26389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:17.412448 26389 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34131
I20260812 11:15:17.412482 26526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34131 every 8 connection(s)
I20260812 11:15:17.413447 26389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:17.416666 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:17.422065 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: Bootstrap starting.
I20260812 11:15:17.422443 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26389
I20260812 11:15:17.422588 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:17.422926 23943 external_mini_cluster.cc:1428] Setting key 0ff3600faa4275e8e3b7b139a85bfe1d
I20260812 11:15:17.424901 26529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:17.425640 26529 log.cc:824] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:17.427484 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: No bootstrap required, opened a new log
I20260812 11:15:17.431113 26529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } }
I20260812 11:15:17.431365 26529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:17.431429 26529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708e6d5de4da49d5887a19959a95658f, State: Initialized, Role: FOLLOWER
I20260812 11:15:17.431911 26529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } }
I20260812 11:15:17.432066 26529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:17.432149 26529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:17.432262 26529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:17.432871 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.424267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.23.97.254:34131)
0812 11:15:17.424927 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.424942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.425013 (+ 71us) server_negotiation.cc:407] Connection header received
0812 11:15:17.425780 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.425808 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.426212 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.426554 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:17.427001 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.428041 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.429056 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.429560 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.429760 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.430749 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:17.430755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:17.430769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:17.431153 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:17.431211 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:17.431235 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:17.431365 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.431547 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.431826 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":121,"threads_started":1}
I20260812 11:15:17.433424 26529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } }
I20260812 11:15:17.433777 26529 leader_election.cc:302] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708e6d5de4da49d5887a19959a95658f; no voters:
I20260812 11:15:17.434120 26529 leader_election.cc:288] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:17.434203 26535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:17.434439 26535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [term 1 LEADER]: Becoming Leader. State: Replica: 708e6d5de4da49d5887a19959a95658f, State: Running, Role: LEADER
I20260812 11:15:17.434755 26535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } }
I20260812 11:15:17.435508 26529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:17.436739 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708e6d5de4da49d5887a19959a95658f. Latest consensus state: current_term: 1 leader_uuid: "708e6d5de4da49d5887a19959a95658f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } } }
I20260812 11:15:17.436870 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:17.436926 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708e6d5de4da49d5887a19959a95658f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708e6d5de4da49d5887a19959a95658f" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34131 } } }
I20260812 11:15:17.437089 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:17.437288 26548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:17.440351 26548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:17.446015 26548 catalog_manager.cc:1347] Generated new cluster ID: fa2b4c1ea5a04ca4b0eb9a9691ad00c9
I20260812 11:15:17.446116 26548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:17.465689 26548 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:17.466722 26548 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:17.470820 26548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: Generated new TSK 0
I20260812 11:15:17.471657 26548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:17.474741 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--builtin_ntp_servers=127.23.97.212:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:17.590903 26564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.591197 26564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.591307 26564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.595443 26564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.595517 26564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.595602 26564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:17.600430 26564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:17.601506 26564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.602994 26564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:17.612733 26569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.612810 26580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.612910 26575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.613050 26564 server_base.cc:1034] running on GCE node
I20260812 11:15:17.613459 26564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:17.614120 26564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:17.615299 26564 hybrid_clock.cc:648] HybridClock initialized: now 1786533317615256 us; error 47 us; skew 500 ppm
I20260812 11:15:17.617216 26564 webserver.cc:460] Webserver started at http://127.23.97.193:37779/ using document root <none> and password file <none>
I20260812 11:15:17.617775 26564 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:17.617836 26564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:17.618009 26564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:17.620110 26564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "1bf52e4e6e4e4bce92f5be926251b8ff"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5fa8cdbf8f2345422a2dc311df1f42ad"
server_key_iv: "fcdeec72cfb497171f3bdceecf8a5b7a"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.620720 26564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "1bf52e4e6e4e4bce92f5be926251b8ff"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5fa8cdbf8f2345422a2dc311df1f42ad"
server_key_iv: "fcdeec72cfb497171f3bdceecf8a5b7a"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.624495 26564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:17.626893 26589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.628075 26564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:17.628170 26564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "1bf52e4e6e4e4bce92f5be926251b8ff"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5fa8cdbf8f2345422a2dc311df1f42ad"
server_key_iv: "fcdeec72cfb497171f3bdceecf8a5b7a"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.628261 26564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:17.642810 26564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:17.643554 26564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:17.643771 26564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:17.644445 26564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:17.645565 26564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:17.645639 26564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.645709 26564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:17.645740 26564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.654513 26564 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:42841
I20260812 11:15:17.654543 26781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:42841 every 8 connection(s)
I20260812 11:15:17.655411 26564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:17.662569 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26564
I20260812 11:15:17.662737 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:17.663066 23943 external_mini_cluster.cc:1428] Setting key 7582e795a5096f680007e93bf5356887
I20260812 11:15:17.665198 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--builtin_ntp_servers=127.23.97.212:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:17.665241 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.657371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:44141 (local address 127.23.97.254:34131)
0812 11:15:17.657510 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.657514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.658598 (+ 1084us) server_negotiation.cc:407] Connection header received
0812 11:15:17.660056 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.660060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.660121 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.660201 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:17.661164 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.661742 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.663045 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.663355 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.663472 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.664716 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:17.664720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:17.664723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:17.664736 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:17.664761 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:17.664776 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:17.664865 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.665038 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.665102 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:15:17.666100 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.657783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34131 (local address 127.23.97.193:44141)
0812 11:15:17.658442 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.658483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.659825 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.660424 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.660440 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.660528 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:17.661007 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.661020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.661891 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.661897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.662916 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.662927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.663525 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.663531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.663710 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.664451 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:17.664453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:17.664496 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:17.664563 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:17.664880 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:17.664892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:17.664898 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:17.665038 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 11:15:17.665339 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260812 11:15:17.667426 26782 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34131
I20260812 11:15:17.667670 26782 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:17.668203 26782 heartbeater.cc:511] Master 127.23.97.254:34131 requested a full tablet report, sending...
I20260812 11:15:17.669828 26450 ts_manager.cc:194] Registered new tserver with Master: 1bf52e4e6e4e4bce92f5be926251b8ff (127.23.97.193:42841)
I20260812 11:15:17.671025 26450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.193:44141
W20260812 11:15:17.788609 26787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.788864 26787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.788902 26787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.792904 26787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.792954 26787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.793006 26787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:17.798075 26787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:17.799098 26787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.800521 26787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:17.810667 26798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.810758 26794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:17.810667 26795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:17.811787 26787 server_base.cc:1034] running on GCE node
I20260812 11:15:17.812225 26787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:17.812959 26787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:17.814152 26787 hybrid_clock.cc:648] HybridClock initialized: now 1786533317814141 us; error 42 us; skew 500 ppm
I20260812 11:15:17.816454 26787 webserver.cc:460] Webserver started at http://127.23.97.194:33139/ using document root <none> and password file <none>
I20260812 11:15:17.817116 26787 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:17.817196 26787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:17.817420 26787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:17.819634 26787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "14ce10c8df274847a00a505559cd6a6a"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5ed129df21672b93d616272453f785e6"
server_key_iv: "56db6af0e9f21b7dee9fd9554888c128"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.820209 26787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "14ce10c8df274847a00a505559cd6a6a"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5ed129df21672b93d616272453f785e6"
server_key_iv: "56db6af0e9f21b7dee9fd9554888c128"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.823982 26787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:15:17.826553 26813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.827813 26787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:17.827950 26787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "14ce10c8df274847a00a505559cd6a6a"
format_stamp: "Formatted at 2026-08-12 11:15:17 on dist-test-slave-zpvx"
server_key: "5ed129df21672b93d616272453f785e6"
server_key_iv: "56db6af0e9f21b7dee9fd9554888c128"
server_key_version: "encryptionkey@0"
I20260812 11:15:17.828060 26787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:17.848670 26787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:17.849385 26787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:17.849593 26787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:17.850199 26787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:17.851158 26787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:17.851233 26787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.851316 26787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:17.851361 26787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:17.860586 26787 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:36249
I20260812 11:15:17.860630 27019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:36249 every 8 connection(s)
I20260812 11:15:17.861476 26787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:17.863180 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 26787
I20260812 11:15:17.863339 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:17.863658 23943 external_mini_cluster.cc:1428] Setting key 74fb03f50b4d01b9fc3c0d0e79ddafcc
I20260812 11:15:17.865748 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--builtin_ntp_servers=127.23.97.212:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:17.870978 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.863350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:44671 (local address 127.23.97.254:34131)
0812 11:15:17.863496 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:17.863500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:17.864689 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 11:15:17.866130 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:17.866134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:17.866189 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:17.866299 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:17.867203 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.867733 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.868966 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.869201 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.869320 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.870468 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:17.870472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:17.870474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:17.870495 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:17.870513 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:17.870520 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:17.870602 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:17.870768 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 11:15:17.870838 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:15:17.871857 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:17.863732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34131 (local address 127.23.97.194:44671)
0812 11:15:17.864526 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:17.864580 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:17.865887 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:17.866482 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:17.866489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:17.866573 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:17.867043 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.867056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.867875 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.867881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.868835 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:17.868846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:17.869372 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:17.869376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:17.869552 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:17.870195 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:17.870197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:17.870237 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:17.870307 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:17.870605 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:17.870617 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:17.870626 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:15:17.870765 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:15:17.871044 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
I20260812 11:15:17.873060 27022 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34131
I20260812 11:15:17.873323 27022 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:17.873845 27022 heartbeater.cc:511] Master 127.23.97.254:34131 requested a full tablet report, sending...
I20260812 11:15:17.874970 26450 ts_manager.cc:194] Registered new tserver with Master: 14ce10c8df274847a00a505559cd6a6a (127.23.97.194:36249)
I20260812 11:15:17.875846 26450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.194:44671
W20260812 11:15:17.982053 27027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:17.982311 27027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:17.982350 27027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:17.986369 27027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:17.986421 27027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:17.986474 27027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:17.991154 27027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:17.992216 27027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:17.993628 27027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:18.003381 27034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:18.003400 27035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:18.003384 27038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:18.003808 27027 server_base.cc:1034] running on GCE node
I20260812 11:15:18.004208 27027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:18.004880 27027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:18.006039 27027 hybrid_clock.cc:648] HybridClock initialized: now 1786533318006025 us; error 25 us; skew 500 ppm
I20260812 11:15:18.007872 27027 webserver.cc:460] Webserver started at http://127.23.97.195:36087/ using document root <none> and password file <none>
I20260812 11:15:18.008507 27027 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:18.008576 27027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:18.008783 27027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:18.010854 27027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "316886565d6c4056ac5703024bf698de"
format_stamp: "Formatted at 2026-08-12 11:15:18 on dist-test-slave-zpvx"
server_key: "ada85d160961b6215086168509cbdec6"
server_key_iv: "a351ae29bb395859a94517b6677ed1f1"
server_key_version: "encryptionkey@0"
I20260812 11:15:18.011421 27027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "316886565d6c4056ac5703024bf698de"
format_stamp: "Formatted at 2026-08-12 11:15:18 on dist-test-slave-zpvx"
server_key: "ada85d160961b6215086168509cbdec6"
server_key_iv: "a351ae29bb395859a94517b6677ed1f1"
server_key_version: "encryptionkey@0"
I20260812 11:15:18.015304 27027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 11:15:18.018148 27052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:18.019320 27027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:18.019454 27027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "316886565d6c4056ac5703024bf698de"
format_stamp: "Formatted at 2026-08-12 11:15:18 on dist-test-slave-zpvx"
server_key: "ada85d160961b6215086168509cbdec6"
server_key_iv: "a351ae29bb395859a94517b6677ed1f1"
server_key_version: "encryptionkey@0"
I20260812 11:15:18.019563 27027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:18.031255 27027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:18.031948 27027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:18.032133 27027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:18.032804 27027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:18.033715 27027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:18.033785 27027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:18.033849 27027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:18.033878 27027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:18.042979 27027 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:44761
I20260812 11:15:18.042989 27243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:44761 every 8 connection(s)
I20260812 11:15:18.043898 27027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:18.053242 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.045735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:60535 (local address 127.23.97.254:34131)
0812 11:15:18.045871 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.045880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.046850 (+ 970us) server_negotiation.cc:407] Connection header received
0812 11:15:18.048341 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.048345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.048434 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.048517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:18.049398 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.049944 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.051092 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.051323 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.051432 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.052705 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:18.052709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:18.052712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:18.052728 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:18.052751 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:18.052761 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:18.052848 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.053033 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.053095 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:18.053402 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27027
I20260812 11:15:18.053514 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:18.053786 23943 external_mini_cluster.cc:1428] Setting key 8782773c234b9c0b7aac3caf23e1f4ec
I20260812 11:15:18.054174 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.046069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34131 (local address 127.23.97.195:60535)
0812 11:15:18.046703 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.046741 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.048071 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.048681 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.048688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.048770 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:18.049252 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.049266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.050077 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.050085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.050954 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.050963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.051482 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.051486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.051643 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.052331 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:18.052333 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:18.052429 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:18.052526 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:18.052873 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:18.052882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:18.052888 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.053027 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.053286 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260812 11:15:18.055375 27245 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34131
I20260812 11:15:18.055574 27245 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:18.056090 27245 heartbeater.cc:511] Master 127.23.97.254:34131 requested a full tablet report, sending...
I20260812 11:15:18.057080 26450 ts_manager.cc:194] Registered new tserver with Master: 316886565d6c4056ac5703024bf698de (127.23.97.195:44761)
I20260812 11:15:18.057641 26450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.195:60535
I20260812 11:15:18.067742 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:18.073616 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.069907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.23.97.254:34131)
0812 11:15:18.070039 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.070043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.070254 (+ 211us) server_negotiation.cc:407] Connection header received
0812 11:15:18.070325 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.070327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.070364 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.070430 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:18.070694 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.071178 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.072022 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.072218 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.072305 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.073204 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:18.073206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:18.073208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:18.073219 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:18.073252 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:18.073259 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:18.073354 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.073480 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.073511 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:18.076335 26450 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:18.079298 26450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:18.094359 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.088520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:44761 (local address 127.0.0.1:43260)
0812 11:15:18.089689 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.089701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.089817 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.090405 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.090408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.090439 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:18.090585 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.090591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091571 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.091592 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.092839 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.092850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.093775 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.093779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.093882 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.093929 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.094048 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.094183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":57,"spinlock_wait_cycles":7808,"thread_start_us":112,"threads_started":1}
I20260812 11:15:18.094400 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.089038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.23.97.195:44761)
0812 11:15:18.089503 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.089515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.089805 (+ 290us) server_negotiation.cc:407] Connection header received
0812 11:15:18.089939 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.089943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.090097 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.090260 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:18.090707 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091449 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.092969 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.093647 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.093755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.093955 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.094095 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.094204 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":157,"threads_started":1}
I20260812 11:15:18.095109 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.089038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.23.97.193:42841)
0812 11:15:18.089723 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.089736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.089762 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:15:18.089852 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.089858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.090071 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.090240 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:18.090534 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091748 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.093438 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.094097 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.094190 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.094358 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.094862 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.094956 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":121,"threads_started":1}
I20260812 11:15:18.095101 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.088217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:42841 (local address 127.0.0.1:53238)
0812 11:15:18.089512 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.089528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.089623 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.090250 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.090256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.090279 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:18.090412 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.090419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091898 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.091904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.093281 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.093293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.094581 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.094589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.094692 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.094716 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.094829 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.094966 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":63,"thread_start_us":66,"threads_started":1}
I20260812 11:15:18.095522 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.088351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.23.97.194:36249)
0812 11:15:18.089865 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.089877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.089897 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:15:18.089961 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.089965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.090107 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.090241 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:18.090726 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091843 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.093421 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.094232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.094339 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.095194 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.095257 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.095351 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":130,"threads_started":1}
I20260812 11:15:18.096650 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.088196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36249 (local address 127.0.0.1:51638)
0812 11:15:18.089056 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.089177 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.089307 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.090407 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.090409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.090439 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:18.090585 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.090595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.091956 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.091961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.093288 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.093293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.094760 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.094764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.094850 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.094929 (+ 79us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.094995 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.095063 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":138,"spinlock_wait_cycles":10240,"thread_start_us":130,"threads_started":1}
I20260812 11:15:18.097728 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5da1cb353c4c829b90ee646875b974 (DEFAULT_TABLE table=test-table [id=da65b20234c24fe692ed6ff57d85a090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:18.098930 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5da1cb353c4c829b90ee646875b974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:18.098961 26672 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5da1cb353c4c829b90ee646875b974 (DEFAULT_TABLE table=test-table [id=da65b20234c24fe692ed6ff57d85a090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:18.099884 26672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5da1cb353c4c829b90ee646875b974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:18.099768 26896 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5da1cb353c4c829b90ee646875b974 (DEFAULT_TABLE table=test-table [id=da65b20234c24fe692ed6ff57d85a090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:18.100719 26896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5da1cb353c4c829b90ee646875b974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:18.105888 27281 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Bootstrap starting.
I20260812 11:15:18.107486 27283 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Bootstrap starting.
I20260812 11:15:18.108043 27282 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Bootstrap starting.
I20260812 11:15:18.108603 27281 tablet_bootstrap.cc:654] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:18.109360 27281 log.cc:824] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:18.109699 27283 tablet_bootstrap.cc:654] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:18.110440 27282 tablet_bootstrap.cc:654] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:18.110538 27283 log.cc:824] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:18.111251 27282 log.cc:824] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:18.111819 27281 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: No bootstrap required, opened a new log
I20260812 11:15:18.112026 27281 ts_tablet_manager.cc:1397] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:18.112723 27283 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: No bootstrap required, opened a new log
I20260812 11:15:18.112910 27283 ts_tablet_manager.cc:1397] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260812 11:15:18.113487 27282 tablet_bootstrap.cc:492] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: No bootstrap required, opened a new log
I20260812 11:15:18.113682 27282 ts_tablet_manager.cc:1397] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:18.115573 27281 raft_consensus.cc:348] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.115828 27281 raft_consensus.cc:374] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:18.115882 27281 raft_consensus.cc:729] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf52e4e6e4e4bce92f5be926251b8ff, State: Initialized, Role: FOLLOWER
I20260812 11:15:18.116343 27281 consensus_queue.cc:260] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.117223 27283 raft_consensus.cc:348] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.117473 27283 raft_consensus.cc:374] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:18.117537 27283 raft_consensus.cc:729] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ce10c8df274847a00a505559cd6a6a, State: Initialized, Role: FOLLOWER
I20260812 11:15:18.117715 27281 ts_tablet_manager.cc:1428] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:18.117820 27282 raft_consensus.cc:348] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.117964 27283 consensus_queue.cc:260] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.118053 27282 raft_consensus.cc:374] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:18.118099 27282 raft_consensus.cc:729] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316886565d6c4056ac5703024bf698de, State: Initialized, Role: FOLLOWER
I20260812 11:15:18.118187 26782 heartbeater.cc:503] Master 127.23.97.254:34131 was elected leader, sending a full tablet report...
I20260812 11:15:18.118620 27022 heartbeater.cc:503] Master 127.23.97.254:34131 was elected leader, sending a full tablet report...
I20260812 11:15:18.118924 27283 ts_tablet_manager.cc:1428] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260812 11:15:18.119529 27282 consensus_queue.cc:260] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.120457 27245 heartbeater.cc:503] Master 127.23.97.254:34131 was elected leader, sending a full tablet report...
I20260812 11:15:18.120527 27282 ts_tablet_manager.cc:1428] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260812 11:15:18.156663 26783 tablet.cc:2369] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:18.157126 27302 raft_consensus.cc:482] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:18.157364 27302 raft_consensus.cc:504] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.158605 27302 leader_election.cc:288] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ce10c8df274847a00a505559cd6a6a (127.23.97.194:36249), 1bf52e4e6e4e4bce92f5be926251b8ff (127.23.97.193:42841)
I20260812 11:15:18.162279 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.159064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:34913 (local address 127.23.97.194:36249)
0812 11:15:18.159209 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.159213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.159238 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:15:18.159285 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.159287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.159330 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.159430 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:18.159734 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.160254 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.161311 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.161707 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.161840 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.161893 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.162076 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.162145 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:15:18.162279 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.158804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:36249 (local address 127.23.97.195:34913)
0812 11:15:18.158960 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.158976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.159088 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.159444 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.159447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.159466 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:18.159596 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.159608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.160395 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.160398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.161175 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.161185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.161833 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.161837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.161939 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.161966 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.162058 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.162145 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 11:15:18.162551 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.158928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:42841 (local address 127.23.97.195:53475)
0812 11:15:18.159319 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.159329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:18.159446 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:18.159720 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:18.159723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:18.159739 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:18.159871 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.159877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.160680 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.160685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.161454 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:18.161462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.162110 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.162114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:18.162216 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.162245 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:15:18.162341 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 11:15:18.162411 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":135,"threads_started":1}
I20260812 11:15:18.162539 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.159022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:53475 (local address 127.23.97.193:42841)
0812 11:15:18.159158 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.159165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.159423 (+ 258us) server_negotiation.cc:407] Connection header received
0812 11:15:18.159569 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.159572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.159621 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.159708 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:18.159998 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.160565 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.161592 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.161969 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.162109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.162157 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.162358 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.162430 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:15:18.162992 26939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5da1cb353c4c829b90ee646875b974" candidate_uuid: "316886565d6c4056ac5703024bf698de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ce10c8df274847a00a505559cd6a6a" is_pre_election: true
I20260812 11:15:18.163105 26711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5da1cb353c4c829b90ee646875b974" candidate_uuid: "316886565d6c4056ac5703024bf698de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" is_pre_election: true
I20260812 11:15:18.163355 26939 raft_consensus.cc:2454] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316886565d6c4056ac5703024bf698de in term 0.
I20260812 11:15:18.163415 26711 raft_consensus.cc:2454] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316886565d6c4056ac5703024bf698de in term 0.
I20260812 11:15:18.163920 27060 leader_election.cc:302] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf52e4e6e4e4bce92f5be926251b8ff, 316886565d6c4056ac5703024bf698de; no voters:
I20260812 11:15:18.164294 27302 raft_consensus.cc:2781] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:18.164449 27302 raft_consensus.cc:482] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:18.164515 27302 raft_consensus.cc:3037] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:18.165850 27302 raft_consensus.cc:504] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.166209 27302 leader_election.cc:288] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [CANDIDATE]: Term 1 election: Requested vote from peers 14ce10c8df274847a00a505559cd6a6a (127.23.97.194:36249), 1bf52e4e6e4e4bce92f5be926251b8ff (127.23.97.193:42841)
I20260812 11:15:18.166563 26939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5da1cb353c4c829b90ee646875b974" candidate_uuid: "316886565d6c4056ac5703024bf698de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ce10c8df274847a00a505559cd6a6a"
I20260812 11:15:18.166731 26711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5da1cb353c4c829b90ee646875b974" candidate_uuid: "316886565d6c4056ac5703024bf698de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff"
I20260812 11:15:18.166725 26939 raft_consensus.cc:3037] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:18.166874 26711 raft_consensus.cc:3037] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:18.167874 26939 raft_consensus.cc:2454] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316886565d6c4056ac5703024bf698de in term 1.
I20260812 11:15:18.167958 26711 raft_consensus.cc:2454] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316886565d6c4056ac5703024bf698de in term 1.
I20260812 11:15:18.168223 27061 leader_election.cc:302] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ce10c8df274847a00a505559cd6a6a, 316886565d6c4056ac5703024bf698de; no voters:
I20260812 11:15:18.168423 27302 raft_consensus.cc:2781] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:18.168678 27302 raft_consensus.cc:686] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 1 LEADER]: Becoming Leader. State: Replica: 316886565d6c4056ac5703024bf698de, State: Running, Role: LEADER
I20260812 11:15:18.169016 27302 consensus_queue.cc:237] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } }
I20260812 11:15:18.171950 26438 catalog_manager.cc:5495] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de reported cstate change: term changed from 0 to 1, leader changed from <none> to 316886565d6c4056ac5703024bf698de (127.23.97.195). New cstate: current_term: 1 leader_uuid: "316886565d6c4056ac5703024bf698de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316886565d6c4056ac5703024bf698de" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260812 11:15:18.192988 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.189372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.23.97.195:44761)
0812 11:15:18.189501 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:18.189504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:18.189519 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:15:18.189665 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:18.189668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:18.189723 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:18.189812 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:18.190062 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.190556 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.191465 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:18.191667 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:18.191746 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:18.191997 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:18.192478 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:18.192598 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:18.192752 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 11:15:18.192810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:18.198853 26711 raft_consensus.cc:1261] T 0f5da1cb353c4c829b90ee646875b974 P 1bf52e4e6e4e4bce92f5be926251b8ff [term 1 FOLLOWER]: Refusing update from remote peer 316886565d6c4056ac5703024bf698de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:18.198863 26939 raft_consensus.cc:1261] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 1 FOLLOWER]: Refusing update from remote peer 316886565d6c4056ac5703024bf698de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:18.199582 27302 consensus_queue.cc:1035] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf52e4e6e4e4bce92f5be926251b8ff" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:18.199771 27305 consensus_queue.cc:1035] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ce10c8df274847a00a505559cd6a6a" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:18.210405 27309 mvcc.cc:205] Tried to move back new op lower bound from 7317640471335677952 to 7317640471223365632. Current Snapshot: MvccSnapshot[applied={T|T < 7317640471335677952}]
I20260812 11:15:18.216145 26450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:18.216321 26450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:18.218802 26450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 708e6d5de4da49d5887a19959a95658f: Sending DeleteTablet for 3 replicas of tablet 0f5da1cb353c4c829b90ee646875b974
I20260812 11:15:18.219530 26896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5da1cb353c4c829b90ee646875b974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:18 UTC) from {username='slave'} at 127.0.0.1:51638
I20260812 11:15:18.219646 27142 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5da1cb353c4c829b90ee646875b974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:18 UTC) from {username='slave'} at 127.0.0.1:43260
I20260812 11:15:18.220068 27330 tablet_replica.cc:330] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: stopping tablet replica
I20260812 11:15:18.220407 27330 raft_consensus.cc:2229] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:18.220400 26672 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5da1cb353c4c829b90ee646875b974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:18 UTC) from {username='slave'} at 127.0.0.1:53238
I20260812 11:15:18.220723 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26564
I20260812 11:15:18.220850 27332 tablet_replica.cc:330] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: stopping tablet replica
I20260812 11:15:18.221112 27332 raft_consensus.cc:2229] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:18.221508 27332 raft_consensus.cc:2258] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:18.239065 27330 raft_consensus.cc:2258] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:18.242693 27332 ts_tablet_manager.cc:1905] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:18.242707 27330 ts_tablet_manager.cc:1905] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:18.245323 27332 ts_tablet_manager.cc:1918] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:18.245445 27332 log.cc:1196] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/0f5da1cb353c4c829b90ee646875b974
I20260812 11:15:18.245808 27332 ts_tablet_manager.cc:1939] T 0f5da1cb353c4c829b90ee646875b974 P 316886565d6c4056ac5703024bf698de: Deleting consensus metadata
I20260812 11:15:18.246294 27330 ts_tablet_manager.cc:1918] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:18.246416 27330 log.cc:1196] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/0f5da1cb353c4c829b90ee646875b974
I20260812 11:15:18.246730 27330 ts_tablet_manager.cc:1939] T 0f5da1cb353c4c829b90ee646875b974 P 14ce10c8df274847a00a505559cd6a6a: Deleting consensus metadata
I20260812 11:15:18.247210 26419 catalog_manager.cc:4841] TS 316886565d6c4056ac5703024bf698de (127.23.97.195:44761): tablet 0f5da1cb353c4c829b90ee646875b974 (table test-table [id=da65b20234c24fe692ed6ff57d85a090]) successfully deleted
I20260812 11:15:18.248088 26423 catalog_manager.cc:4841] TS 14ce10c8df274847a00a505559cd6a6a (127.23.97.194:36249): tablet 0f5da1cb353c4c829b90ee646875b974 (table test-table [id=da65b20234c24fe692ed6ff57d85a090]) successfully deleted
W20260812 11:15:18.250229 26422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:42841 (error 108)
I20260812 11:15:18.250551 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26787
I20260812 11:15:18.255429 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:18.255081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:42841 (local address 127.0.0.1:53244)
0812 11:15:18.255252 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:18.255338 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:15:18.255640 26422 catalog_manager.cc:4584] TS 1bf52e4e6e4e4bce92f5be926251b8ff (127.23.97.193:42841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f5da1cb353c4c829b90ee646875b974: Network error: Client connection negotiation failed: client connection to 127.23.97.193:42841: connect: Connection refused (error 111)
I20260812 11:15:18.260313 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27027
I20260812 11:15:18.274582 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 26389
2026-08-12T11:15:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:18 dist-test-slave-zpvx krb5kdc[27343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:18 dist-test-slave-zpvx krb5kdc[27343](info): set up 2 sockets
Aug 12 11:15:18 dist-test-slave-zpvx krb5kdc[27343](info): commencing operation
krb5kdc: starting...
W20260812 11:15:19.586786 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:19 dist-test-slave-zpvx krb5kdc[27343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:19.713413 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34829
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:19.824427 27363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:19.824744 27363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:19.824819 27363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:19.828750 27363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:19.828837 27363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:19.828876 27363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:19.828913 27363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:19.828949 27363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:19.834084 27363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:19.835141 27363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:19.836177 27363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:19.844652 27372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:19.844780 27363 server_base.cc:1034] running on GCE node
W20260812 11:15:19.844727 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:19.844722 27374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:19.845448 27363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:19.846495 27363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:19.847684 27363 hybrid_clock.cc:648] HybridClock initialized: now 1786533319847667 us; error 32 us; skew 500 ppm
Aug 12 11:15:19 dist-test-slave-zpvx krb5kdc[27343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533319, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:19.850498 27363 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:19.851300 27363 webserver.cc:460] Webserver started at http://127.23.97.254:36273/ using document root <none> and password file <none>
I20260812 11:15:19.851809 27363 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:19.851868 27363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:19.852027 27363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:19.854044 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "2232f29ad061433faee1dcb07629e629"
format_stamp: "Formatted at 2026-08-12 11:15:19 on dist-test-slave-zpvx"
server_key: "9467687fba84de8b148dd7adb2e67ac3"
server_key_iv: "f4dc2aa060c1c478aca6dc96bb4a4eab"
server_key_version: "encryptionkey@0"
I20260812 11:15:19.854560 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "2232f29ad061433faee1dcb07629e629"
format_stamp: "Formatted at 2026-08-12 11:15:19 on dist-test-slave-zpvx"
server_key: "9467687fba84de8b148dd7adb2e67ac3"
server_key_iv: "f4dc2aa060c1c478aca6dc96bb4a4eab"
server_key_version: "encryptionkey@0"
I20260812 11:15:19.858225 27363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260812 11:15:19.860623 27384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:19.861893 27363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:15:19.862021 27363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "2232f29ad061433faee1dcb07629e629"
format_stamp: "Formatted at 2026-08-12 11:15:19 on dist-test-slave-zpvx"
server_key: "9467687fba84de8b148dd7adb2e67ac3"
server_key_iv: "f4dc2aa060c1c478aca6dc96bb4a4eab"
server_key_version: "encryptionkey@0"
I20260812 11:15:19.862138 27363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:19.875573 27363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:19.878607 27363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:19.878809 27363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:19.885826 27363 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34829
I20260812 11:15:19.885916 27474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34829 every 8 connection(s)
I20260812 11:15:19.886775 27363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:19.889935 27476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:19.890040 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27363
I20260812 11:15:19.890218 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:19.890587 23943 external_mini_cluster.cc:1428] Setting key be4d425590aef4a13ea7fd8798cc50e9
I20260812 11:15:19.897046 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: Bootstrap starting.
I20260812 11:15:19.899416 27476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: Neither blocks nor log segments found. Creating new log.
Aug 12 11:15:19 dist-test-slave-zpvx krb5kdc[27343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:19.900283 27476 log.cc:824] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:19.902521 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: No bootstrap required, opened a new log
I20260812 11:15:19.905048 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:19.892152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.23.97.254:34829)
0812 11:15:19.892676 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:19.892690 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:19.892736 (+ 46us) server_negotiation.cc:407] Connection header received
0812 11:15:19.893568 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:19.893587 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:19.893919 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:19.894273 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:19.895120 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:19.896083 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:19.897047 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:19.897578 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:19.897765 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:19.900434 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:19.900442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:19.900463 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:19.900496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:19.902256 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:19.902714 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:19.902719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:19.902724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:19.902832 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:19.903121 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:19.903124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:19.903126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:19.903507 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:19.903721 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:19.903988 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 11:15:19.904287 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":178,"threads_started":1}
I20260812 11:15:19.906733 27476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } }
I20260812 11:15:19.907017 27476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:19.907087 27476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2232f29ad061433faee1dcb07629e629, State: Initialized, Role: FOLLOWER
I20260812 11:15:19.907591 27476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } }
I20260812 11:15:19.907740 27476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:19.907794 27476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:19.907871 27476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:19.908881 27476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } }
I20260812 11:15:19.909209 27476 leader_election.cc:302] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2232f29ad061433faee1dcb07629e629; no voters:
I20260812 11:15:19.909523 27476 leader_election.cc:288] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:19.909621 27487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:19.909868 27487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [term 1 LEADER]: Becoming Leader. State: Replica: 2232f29ad061433faee1dcb07629e629, State: Running, Role: LEADER
I20260812 11:15:19.910184 27487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } }
I20260812 11:15:19.910951 27476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:19.911284 27490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2232f29ad061433faee1dcb07629e629. Latest consensus state: current_term: 1 leader_uuid: "2232f29ad061433faee1dcb07629e629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } } }
I20260812 11:15:19.911504 27490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:19.911742 27488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2232f29ad061433faee1dcb07629e629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2232f29ad061433faee1dcb07629e629" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34829 } } }
I20260812 11:15:19.911854 27488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:19.912115 27492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:19.915264 27492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:19.921677 27492 catalog_manager.cc:1347] Generated new cluster ID: 9abb18cc81ba471fa8957582c4fc3a65
I20260812 11:15:19.921779 27492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:19.927935 27492 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:19.929050 27492 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:19.940320 27492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: Generated new TSK 0
I20260812 11:15:19.940989 27492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.97.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
Entry for principal kudu/127.23.97.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab.
I20260812 11:15:19.975304 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:20.088479 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.088774 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.088853 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.092835 27528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.092918 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.093007 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:20.098120 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.193.keytab
--principal=kudu/127.23.97.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:20.099123 27528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:20.100147 27528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:20.109525 27533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.109591 27528 server_base.cc:1034] running on GCE node
W20260812 11:15:20.109549 27534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.109555 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.110229 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:20.110868 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:20.112082 27528 hybrid_clock.cc:648] HybridClock initialized: now 1786533320112041 us; error 52 us; skew 500 ppm
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:20.115459 27528 init.cc:377] Logged in from keytab as kudu/127.23.97.193@KRBTEST.COM (short username kudu)
I20260812 11:15:20.116354 27528 webserver.cc:460] Webserver started at http://127.23.97.193:33629/ using document root <none> and password file <none>
I20260812 11:15:20.117110 27528 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:20.117213 27528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:20.117431 27528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:20.119486 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "0db6b9c1a79a47229b37826a6669b193"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "d4d206c6e7fcf5bf6fb76fe8c4e57d63"
server_key_iv: "347a46e98c6be13525f87051b2598ca7"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.120077 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "0db6b9c1a79a47229b37826a6669b193"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "d4d206c6e7fcf5bf6fb76fe8c4e57d63"
server_key_iv: "347a46e98c6be13525f87051b2598ca7"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.123998 27528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260812 11:15:20.126585 27545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.127851 27528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:20.127990 27528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "0db6b9c1a79a47229b37826a6669b193"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "d4d206c6e7fcf5bf6fb76fe8c4e57d63"
server_key_iv: "347a46e98c6be13525f87051b2598ca7"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.128110 27528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:20.160677 27528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:20.164317 27528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:20.164558 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:20.165252 27528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:20.166177 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:20.166246 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.166313 27528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:20.166365 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.175272 27528 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:36383
I20260812 11:15:20.175298 27735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:36383 every 8 connection(s)
I20260812 11:15:20.176163 27528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:20.182646 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27528
I20260812 11:15:20.182844 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:20.183168 23943 external_mini_cluster.cc:1428] Setting key fef82ceccdd6df95459d45c2eecf5749
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.193@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:20.191131 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.178068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:41107 (local address 127.23.97.254:34829)
0812 11:15:20.178213 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.178217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.179074 (+ 857us) server_negotiation.cc:407] Connection header received
0812 11:15:20.180522 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.180525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.180588 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.180701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:20.181877 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.182429 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.183740 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.184053 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.184164 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.187135 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:20.187140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:20.187146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:20.187181 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:20.188899 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.189460 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.189464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.189466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.189555 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.189979 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.189982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.189984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.190175 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:20.190288 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.190807 (+ 519us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.190951 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:15:20.191955 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.178480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34829 (local address 127.23.97.193:41107)
0812 11:15:20.178926 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.178968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.180232 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.180885 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.180892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.181238 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:20.181719 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.181733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.182563 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.182567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.183615 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.183626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.184232 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.184236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.184483 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.185182 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:20.185199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:20.186952 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:20.189033 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.189039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.189059 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.189339 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.189664 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.189668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.189670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.189872 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.190289 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:20.190297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:20.190551 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.190747 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.191106 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260812 11:15:20.193416 27737 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34829
I20260812 11:15:20.193691 27737 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.97.194@KRBTEST.COM; defaulting to no policy
I20260812 11:15:20.194319 27737 heartbeater.cc:511] Master 127.23.97.254:34829 requested a full tablet report, sending...
I20260812 11:15:20.195806 27413 ts_manager.cc:194] Registered new tserver with Master: 0db6b9c1a79a47229b37826a6669b193 (127.23.97.193:36383)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.194@KRBTEST.COM" created.
I20260812 11:15:20.197114 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.193@KRBTEST.COM'} at 127.23.97.193:41107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
Entry for principal kudu/127.23.97.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab.
I20260812 11:15:20.213459 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:20.327455 27750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.327739 27750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.327816 27750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.331976 27750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.332051 27750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.332139 27750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:20.337437 27750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.194.keytab
--principal=kudu/127.23.97.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:20.338464 27750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:20.339461 27750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:20.348846 27768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.348881 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.348897 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.349211 27750 server_base.cc:1034] running on GCE node
I20260812 11:15:20.349629 27750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:20.350293 27750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:20.351501 27750 hybrid_clock.cc:648] HybridClock initialized: now 1786533320351480 us; error 39 us; skew 500 ppm
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:20.354336 27750 init.cc:377] Logged in from keytab as kudu/127.23.97.194@KRBTEST.COM (short username kudu)
I20260812 11:15:20.355125 27750 webserver.cc:460] Webserver started at http://127.23.97.194:39137/ using document root <none> and password file <none>
I20260812 11:15:20.355629 27750 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:20.355686 27750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:20.355846 27750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:20.358148 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "c938c5b7b2d140d09d1eabd84e68151b"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "3babcca6b74bb61961aa309bb773230b"
server_key_iv: "3dd9cdea32d1f0400ebeefc229b4724e"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.358666 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "c938c5b7b2d140d09d1eabd84e68151b"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "3babcca6b74bb61961aa309bb773230b"
server_key_iv: "3dd9cdea32d1f0400ebeefc229b4724e"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.362468 27750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:15:20.364912 27782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.366412 27750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:20.366571 27750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "c938c5b7b2d140d09d1eabd84e68151b"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "3babcca6b74bb61961aa309bb773230b"
server_key_iv: "3dd9cdea32d1f0400ebeefc229b4724e"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.366688 27750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:20.380162 27750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:20.383800 27750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:20.384032 27750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:20.384758 27750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:20.385699 27750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:20.385753 27750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.385829 27750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:20.385874 27750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.395985 27750 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:34983
I20260812 11:15:20.396021 27969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:34983 every 8 connection(s)
I20260812 11:15:20.396967 27750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:20.400319 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27750
I20260812 11:15:20.400545 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:20.400930 23943 external_mini_cluster.cc:1428] Setting key 1181e68c9d619c334b801ab19d590921
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.194@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.195@KRBTEST.COM; defaulting to no policy
I20260812 11:15:20.412767 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.398829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:34715 (local address 127.23.97.254:34829)
0812 11:15:20.399006 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.399011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.399839 (+ 828us) server_negotiation.cc:407] Connection header received
0812 11:15:20.401659 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.401662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.401714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.401795 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:20.403054 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.403885 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.405586 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.405977 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.406103 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.408546 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:20.408551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:20.408557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:20.408594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:20.410340 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.410846 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.410852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.410855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.410980 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.411377 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.411381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.411385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.411618 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:20.411744 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.412414 (+ 670us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.412539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:15:20.413535 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.399195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34829 (local address 127.23.97.194:34715)
0812 11:15:20.399684 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.399727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.401420 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.402002 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.402012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.402429 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:20.402904 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.402917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.404018 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.404020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.405442 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.405458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.406135 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.406137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.406276 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.406882 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:20.406898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:20.408308 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:20.410462 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.410467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.410478 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.410723 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.411087 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.411090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.411091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.411224 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.411735 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:20.411740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:20.412081 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.412238 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.412606 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":114,"threads_started":1}
I20260812 11:15:20.414990 27971 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34829
I20260812 11:15:20.415221 27971 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.195@KRBTEST.COM" created.
I20260812 11:15:20.415758 27971 heartbeater.cc:511] Master 127.23.97.254:34829 requested a full tablet report, sending...
I20260812 11:15:20.417002 27413 ts_manager.cc:194] Registered new tserver with Master: c938c5b7b2d140d09d1eabd84e68151b (127.23.97.194:34983)
I20260812 11:15:20.417680 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.194@KRBTEST.COM'} at 127.23.97.194:34715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
Entry for principal kudu/127.23.97.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab.
I20260812 11:15:20.431545 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:20.551013 27976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.551249 27976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.551290 27976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.555089 27976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.555135 27976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.555188 27976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:20.560318 27976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.195.keytab
--principal=kudu/127.23.97.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:20.561317 27976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:20.562292 27976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:20.572160 27989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.572240 27992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:20.572176 27996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:20.572351 27976 server_base.cc:1034] running on GCE node
I20260812 11:15:20.572906 27976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:20.573562 27976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:20.574769 27976 hybrid_clock.cc:648] HybridClock initialized: now 1786533320574754 us; error 34 us; skew 500 ppm
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:20.578078 27976 init.cc:377] Logged in from keytab as kudu/127.23.97.195@KRBTEST.COM (short username kudu)
I20260812 11:15:20.578886 27976 webserver.cc:460] Webserver started at http://127.23.97.195:43651/ using document root <none> and password file <none>
I20260812 11:15:20.579423 27976 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:20.579483 27976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:20.579659 27976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:20.581722 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "71380414ce0d415ba385fc5131ba901a"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "29a6a270f99e6bfd305cac42f4f293bf"
server_key_iv: "145def80c1d9d6f3a74b94259034da00"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.582243 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "71380414ce0d415ba385fc5131ba901a"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "29a6a270f99e6bfd305cac42f4f293bf"
server_key_iv: "145def80c1d9d6f3a74b94259034da00"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.585964 27976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:15:20.588548 28008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.590010 27976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:20.590201 27976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "71380414ce0d415ba385fc5131ba901a"
format_stamp: "Formatted at 2026-08-12 11:15:20 on dist-test-slave-zpvx"
server_key: "29a6a270f99e6bfd305cac42f4f293bf"
server_key_iv: "145def80c1d9d6f3a74b94259034da00"
server_key_version: "encryptionkey@0"
I20260812 11:15:20.590320 27976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:20.599311 27976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:20.602196 27976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:20.602387 27976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:20.602931 27976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:20.603736 27976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:20.603806 27976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.603873 27976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:20.603928 27976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:20.613095 27976 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:41597
I20260812 11:15:20.613184 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:41597 every 8 connection(s)
I20260812 11:15:20.613994 27976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:20.618604 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 27976
I20260812 11:15:20.618777 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:20.619061 23943 external_mini_cluster.cc:1428] Setting key 038c885ad3b441d71a768668ded8b995
Aug 12 11:15:20 dist-test-slave-zpvx krb5kdc[27343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.195@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:20.628041 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.615940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:43099 (local address 127.23.97.254:34829)
0812 11:15:20.616095 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.616099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.616936 (+ 837us) server_negotiation.cc:407] Connection header received
0812 11:15:20.618409 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.618413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.618475 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.618580 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:20.619936 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.620554 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.621814 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.622048 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.622157 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.624790 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:20.624793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:20.624796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:20.624822 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:20.626121 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.626629 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.626633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.626634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.626708 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.627048 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.627051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.627052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.627279 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:20.627364 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.627826 (+ 462us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.627924 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:20.628966 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.616299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34829 (local address 127.23.97.195:43099)
0812 11:15:20.616789 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.616831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.618162 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.618793 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.618800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.619216 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:20.619791 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.619806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.620678 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.620682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.621674 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.621684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.622205 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.622209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.622380 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.622876 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:20.622892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:20.624609 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:20.626259 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.626264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.626275 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.626537 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.626794 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:20.626796 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:20.626798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:20.626934 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:20.627384 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:20.627389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:20.627606 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.627775 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.628017 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":112,"threads_started":1}
I20260812 11:15:20.630142 28196 heartbeater.cc:347] Connected to a master server at 127.23.97.254:34829
I20260812 11:15:20.630388 28196 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:20.630870 28196 heartbeater.cc:511] Master 127.23.97.254:34829 requested a full tablet report, sending...
I20260812 11:15:20.632125 27413 ts_manager.cc:194] Registered new tserver with Master: 71380414ce0d415ba385fc5131ba901a (127.23.97.195:41597)
I20260812 11:15:20.632920 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.97.195@KRBTEST.COM'} at 127.23.97.195:43099
I20260812 11:15:20.633342 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:20.642699 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.635387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.23.97.254:34829)
0812 11:15:20.635508 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.635511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.635673 (+ 162us) server_negotiation.cc:407] Connection header received
0812 11:15:20.635755 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.635757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.635791 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.635855 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:20.636614 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.637098 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.638035 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.638254 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.638332 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.639641 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:20.639644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:20.639646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:20.639666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:20.640997 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.641392 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.641395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.641396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.641468 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:20.641848 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:20.641856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:20.641857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:20.642104 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:20.642226 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.642472 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.642581 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 11:15:20.645486 27413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:20.648360 27413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:20.664341 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.658217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.23.97.194:34983)
0812 11:15:20.658705 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.658718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.658744 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:15:20.659003 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.659010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.659266 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.659436 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:20.659865 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661123 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.662823 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663649 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663847 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.663976 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.664060 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.664174 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":147,"threads_started":1}
I20260812 11:15:20.664358 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.657332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:34983 (local address 127.0.0.1:40252)
0812 11:15:20.658575 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.658631 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.658777 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.659611 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.659616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.659637 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:20.659735 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.659740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661277 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.661281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.662691 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.662702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663788 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.663900 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.663928 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.664025 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.664174 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":21,"thread_start_us":93,"threads_started":1}
I20260812 11:15:20.664427 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.657849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.23.97.195:41597)
0812 11:15:20.658311 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.658323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.659037 (+ 714us) server_negotiation.cc:407] Connection header received
0812 11:15:20.659296 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.659303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.659527 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.659647 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:20.660092 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661288 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.662634 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663523 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663652 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.663824 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.664165 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.664268 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":96,"threads_started":1}
I20260812 11:15:20.664386 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.657502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:41597 (local address 127.0.0.1:51854)
0812 11:15:20.658928 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.658942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.659128 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.659821 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.659824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.659843 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:20.659953 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.659959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661455 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.661458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.662494 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.662505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663879 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.663988 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.664023 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.664105 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.664175 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260812 11:15:20.665431 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.658099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36383 (local address 127.0.0.1:41970)
0812 11:15:20.659389 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.659402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.659481 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.660355 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.660359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.660403 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:20.660498 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.660505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661631 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.661634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.662813 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.662821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663657 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.663769 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.665147 (+ 1378us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.665224 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.665306 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":34,"thread_start_us":1167,"threads_started":1}
I20260812 11:15:20.667829 27859 tablet_service.cc:1467] Processing CreateTablet for tablet 08a25023241f4e49961c1be776bc20ff (DEFAULT_TABLE table=test-table [id=59d9841f90584d4fb6e0f134a129b901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:20.667829 28083 tablet_service.cc:1467] Processing CreateTablet for tablet 08a25023241f4e49961c1be776bc20ff (DEFAULT_TABLE table=test-table [id=59d9841f90584d4fb6e0f134a129b901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:20.668551 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.658987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.23.97.193:36383)
0812 11:15:20.659976 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.659989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.660006 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:15:20.660064 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.660068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.660221 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.660381 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:20.660636 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.661467 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.662940 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.663513 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.663658 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.663826 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.665662 (+ 1836us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.665765 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":103,"threads_started":1}
I20260812 11:15:20.668983 28083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a25023241f4e49961c1be776bc20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:20.668985 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a25023241f4e49961c1be776bc20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:20.671661 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 08a25023241f4e49961c1be776bc20ff (DEFAULT_TABLE table=test-table [id=59d9841f90584d4fb6e0f134a129b901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:20.672767 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a25023241f4e49961c1be776bc20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:20.675832 28225 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Bootstrap starting.
I20260812 11:15:20.676985 28227 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Bootstrap starting.
I20260812 11:15:20.678119 28225 tablet_bootstrap.cc:654] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:20.679198 28227 tablet_bootstrap.cc:654] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:20.679348 28225 log.cc:824] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:20.680565 28227 log.cc:824] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:20.681463 28225 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: No bootstrap required, opened a new log
I20260812 11:15:20.681685 28225 ts_tablet_manager.cc:1397] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:20.682403 28227 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: No bootstrap required, opened a new log
I20260812 11:15:20.682552 28227 ts_tablet_manager.cc:1397] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:20.684674 28232 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: Bootstrap starting.
I20260812 11:15:20.685671 28227 raft_consensus.cc:348] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.685938 28227 raft_consensus.cc:374] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:20.686004 28227 raft_consensus.cc:729] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c938c5b7b2d140d09d1eabd84e68151b, State: Initialized, Role: FOLLOWER
I20260812 11:15:20.686437 28227 consensus_queue.cc:260] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.686934 28232 tablet_bootstrap.cc:654] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:20.687464 27971 heartbeater.cc:503] Master 127.23.97.254:34829 was elected leader, sending a full tablet report...
I20260812 11:15:20.687516 28227 ts_tablet_manager.cc:1428] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 11:15:20.688362 28232 log.cc:824] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:20.688880 28225 raft_consensus.cc:348] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.689316 28225 raft_consensus.cc:374] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:20.689420 28225 raft_consensus.cc:729] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71380414ce0d415ba385fc5131ba901a, State: Initialized, Role: FOLLOWER
I20260812 11:15:20.690014 28225 consensus_queue.cc:260] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.691174 28196 heartbeater.cc:503] Master 127.23.97.254:34829 was elected leader, sending a full tablet report...
I20260812 11:15:20.691473 28225 ts_tablet_manager.cc:1428] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260812 11:15:20.693343 28232 tablet_bootstrap.cc:492] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: No bootstrap required, opened a new log
I20260812 11:15:20.693624 28232 ts_tablet_manager.cc:1397] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 11:15:20.697381 28232 raft_consensus.cc:348] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.697626 28232 raft_consensus.cc:374] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:20.697712 28232 raft_consensus.cc:729] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db6b9c1a79a47229b37826a6669b193, State: Initialized, Role: FOLLOWER
I20260812 11:15:20.698132 28232 consensus_queue.cc:260] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.698796 27737 heartbeater.cc:503] Master 127.23.97.254:34829 was elected leader, sending a full tablet report...
I20260812 11:15:20.699013 28232 ts_tablet_manager.cc:1428] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:15:20.717494 28237 raft_consensus.cc:482] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:20.717746 28237 raft_consensus.cc:504] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.718940 28237 leader_election.cc:288] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c938c5b7b2d140d09d1eabd84e68151b (127.23.97.194:34983), 0db6b9c1a79a47229b37826a6669b193 (127.23.97.193:36383)
I20260812 11:15:20.723583 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.719327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:49931 (local address 127.23.97.194:34983)
0812 11:15:20.719451 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.719456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.719475 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:20.719562 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.719565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.719623 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.719725 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:20.720036 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.720802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.722379 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.722980 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.723107 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.723163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.723365 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.723431 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:20.723584 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.719226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:34983 (local address 127.23.97.195:49931)
0812 11:15:20.719330 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.719347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.719451 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.719742 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.719744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.719764 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:20.719918 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.719927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.720957 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.720960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.722237 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.722248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.723102 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.723105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.723215 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.723241 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.723321 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.723402 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":24}
I20260812 11:15:20.723752 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.719226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36383 (local address 127.23.97.195:50975)
0812 11:15:20.719713 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.719746 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:20.719873 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:20.720148 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:20.720151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:20.720169 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:20.720299 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.720305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.721245 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.721248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.722419 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:20.722427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.723271 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.723277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:20.723374 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.723393 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:15:20.723520 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 11:15:20.723581 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":143,"thread_start_us":144,"threads_started":1}
I20260812 11:15:20.723750 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.719327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:50975 (local address 127.23.97.193:36383)
0812 11:15:20.719452 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.719456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.719853 (+ 397us) server_negotiation.cc:407] Connection header received
0812 11:15:20.719967 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.719972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.720040 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.720115 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:20.720498 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.721134 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.722523 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.723131 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.723249 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.723283 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.723543 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.723637 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:20.724455 27898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a25023241f4e49961c1be776bc20ff" candidate_uuid: "71380414ce0d415ba385fc5131ba901a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c938c5b7b2d140d09d1eabd84e68151b" is_pre_election: true
I20260812 11:15:20.724573 27659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a25023241f4e49961c1be776bc20ff" candidate_uuid: "71380414ce0d415ba385fc5131ba901a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db6b9c1a79a47229b37826a6669b193" is_pre_election: true
I20260812 11:15:20.724790 27898 raft_consensus.cc:2454] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71380414ce0d415ba385fc5131ba901a in term 0.
I20260812 11:15:20.724815 27659 raft_consensus.cc:2454] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71380414ce0d415ba385fc5131ba901a in term 0.
I20260812 11:15:20.725363 28013 leader_election.cc:302] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0db6b9c1a79a47229b37826a6669b193, 71380414ce0d415ba385fc5131ba901a, c938c5b7b2d140d09d1eabd84e68151b; no voters:
I20260812 11:15:20.725660 28237 raft_consensus.cc:2781] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:20.725761 28237 raft_consensus.cc:482] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:20.725826 28237 raft_consensus.cc:3037] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:20.726917 28237 raft_consensus.cc:504] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.727279 28237 leader_election.cc:288] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [CANDIDATE]: Term 1 election: Requested vote from peers c938c5b7b2d140d09d1eabd84e68151b (127.23.97.194:34983), 0db6b9c1a79a47229b37826a6669b193 (127.23.97.193:36383)
I20260812 11:15:20.727694 27659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a25023241f4e49961c1be776bc20ff" candidate_uuid: "71380414ce0d415ba385fc5131ba901a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db6b9c1a79a47229b37826a6669b193"
I20260812 11:15:20.727682 27898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a25023241f4e49961c1be776bc20ff" candidate_uuid: "71380414ce0d415ba385fc5131ba901a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c938c5b7b2d140d09d1eabd84e68151b"
I20260812 11:15:20.727818 27898 raft_consensus.cc:3037] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:20.727841 27659 raft_consensus.cc:3037] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:20.729233 27659 raft_consensus.cc:2454] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71380414ce0d415ba385fc5131ba901a in term 1.
I20260812 11:15:20.729233 27898 raft_consensus.cc:2454] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71380414ce0d415ba385fc5131ba901a in term 1.
I20260812 11:15:20.729578 28011 leader_election.cc:302] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0db6b9c1a79a47229b37826a6669b193, 71380414ce0d415ba385fc5131ba901a, c938c5b7b2d140d09d1eabd84e68151b; no voters:
I20260812 11:15:20.729799 28237 raft_consensus.cc:2781] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:20.729988 28237 raft_consensus.cc:686] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 1 LEADER]: Becoming Leader. State: Replica: 71380414ce0d415ba385fc5131ba901a, State: Running, Role: LEADER
I20260812 11:15:20.730372 28237 consensus_queue.cc:237] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } }
I20260812 11:15:20.733804 27411 catalog_manager.cc:5495] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a reported cstate change: term changed from 0 to 1, leader changed from <none> to 71380414ce0d415ba385fc5131ba901a (127.23.97.195). New cstate: current_term: 1 leader_uuid: "71380414ce0d415ba385fc5131ba901a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71380414ce0d415ba385fc5131ba901a" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 41597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260812 11:15:20.744176 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.741001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.23.97.195:41597)
0812 11:15:20.741126 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:20.741130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:20.741142 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:20.741265 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:20.741267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:20.741322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:20.741411 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:20.741633 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.742125 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.742924 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:20.743129 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:20.743208 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:20.743468 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:20.743864 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:20.743953 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:20.744010 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 11:15:20.744056 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:15:20.750000 27659 raft_consensus.cc:1261] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 1 FOLLOWER]: Refusing update from remote peer 71380414ce0d415ba385fc5131ba901a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:20.750000 27898 raft_consensus.cc:1261] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 1 FOLLOWER]: Refusing update from remote peer 71380414ce0d415ba385fc5131ba901a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:20.750722 28245 consensus_queue.cc:1035] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c938c5b7b2d140d09d1eabd84e68151b" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:20.750895 28237 consensus_queue.cc:1035] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db6b9c1a79a47229b37826a6669b193" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:20.757607 28251 mvcc.cc:205] Tried to move back new op lower bound from 7317640481784803328 to 7317640481714642944. Current Snapshot: MvccSnapshot[applied={T|T < 7317640481784803328}]
I20260812 11:15:20.759707 28253 mvcc.cc:205] Tried to move back new op lower bound from 7317640481784803328 to 7317640481714642944. Current Snapshot: MvccSnapshot[applied={T|T < 7317640481784803328 or (T in {7317640481784803328})}]
I20260812 11:15:20.761569 28252 mvcc.cc:205] Tried to move back new op lower bound from 7317640481784803328 to 7317640481714642944. Current Snapshot: MvccSnapshot[applied={T|T < 7317640481784803328}]
I20260812 11:15:20.766813 27413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:20.766993 27413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:20.769807 27413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2232f29ad061433faee1dcb07629e629: Sending DeleteTablet for 3 replicas of tablet 08a25023241f4e49961c1be776bc20ff
I20260812 11:15:20.770605 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a25023241f4e49961c1be776bc20ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:20 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:40252
I20260812 11:15:20.770668 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a25023241f4e49961c1be776bc20ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:20 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:41970
I20260812 11:15:20.770651 28083 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a25023241f4e49961c1be776bc20ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:20 UTC) from {username='kudu', principal='kudu/127.23.97.254@KRBTEST.COM'} at 127.0.0.1:51854
I20260812 11:15:20.771132 28265 tablet_replica.cc:330] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193: stopping tablet replica
I20260812 11:15:20.771514 28265 raft_consensus.cc:2229] T 08a25023241f4e49961c1be776bc20ff P 0db6b9c1a79a47229b37826a6669b193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:20.771622 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27528
I20260812 11:15:20.771852 28264 tablet_replica.cc:330] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: stopping tablet replica
I20260812 11:15:20.772188 28264 raft_consensus.cc:2229] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:20.772580 28264 raft_consensus.cc:2258] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:20.781231 28264 ts_tablet_manager.cc:1905] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:20.785444 28264 ts_tablet_manager.cc:1918] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:20.785581 28264 log.cc:1196] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/08a25023241f4e49961c1be776bc20ff
I20260812 11:15:20.786062 28264 ts_tablet_manager.cc:1939] T 08a25023241f4e49961c1be776bc20ff P c938c5b7b2d140d09d1eabd84e68151b: Deleting consensus metadata
I20260812 11:15:20.787907 27388 catalog_manager.cc:4841] TS c938c5b7b2d140d09d1eabd84e68151b (127.23.97.194:34983): tablet 08a25023241f4e49961c1be776bc20ff (table test-table [id=59d9841f90584d4fb6e0f134a129b901]) successfully deleted
I20260812 11:15:20.794018 28266 tablet_replica.cc:330] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: stopping tablet replica
I20260812 11:15:20.794525 28266 raft_consensus.cc:2229] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:20.794975 28266 raft_consensus.cc:2258] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:20.797689 28266 ts_tablet_manager.cc:1905] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:20.800568 28266 ts_tablet_manager.cc:1918] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:20.800675 28266 log.cc:1196] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/08a25023241f4e49961c1be776bc20ff
I20260812 11:15:20.801031 28266 ts_tablet_manager.cc:1939] T 08a25023241f4e49961c1be776bc20ff P 71380414ce0d415ba385fc5131ba901a: Deleting consensus metadata
W20260812 11:15:20.802197 27390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:36383 (error 108)
I20260812 11:15:20.802440 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27750
I20260812 11:15:20.802783 27390 catalog_manager.cc:4841] TS 71380414ce0d415ba385fc5131ba901a (127.23.97.195:41597): tablet 08a25023241f4e49961c1be776bc20ff (table test-table [id=59d9841f90584d4fb6e0f134a129b901]) successfully deleted
I20260812 11:15:20.809650 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:20.806442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:36383 (local address 127.0.0.1:41984)
0812 11:15:20.809474 (+ 3032us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:20.809563 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2910}
W20260812 11:15:20.809832 27390 catalog_manager.cc:4584] TS 0db6b9c1a79a47229b37826a6669b193 (127.23.97.193:36383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08a25023241f4e49961c1be776bc20ff: Network error: Client connection negotiation failed: client connection to 127.23.97.193:36383: connect: Connection refused (error 111)
I20260812 11:15:20.819137 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27976
I20260812 11:15:20.837827 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 27363
2026-08-12T11:15:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2571 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T11:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:20Z Disabled control of system clock
I20260812 11:15:20.880582 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:38719
--webserver_interface=127.23.97.254
--webserver_port=0
--builtin_ntp_servers=127.23.97.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:38719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:20.991838 28275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:20.992107 28275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:20.992143 28275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:20.996253 28275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:20.996317 28275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:20.996333 28275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:20.996351 28275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:20.996407 28275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:21.001169 28275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:38719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:21.002306 28275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.003882 28275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.013643 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.013643 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.013787 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.014314 28275 server_base.cc:1034] running on GCE node
I20260812 11:15:21.014953 28275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.016305 28275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.017474 28275 hybrid_clock.cc:648] HybridClock initialized: now 1786533321017440 us; error 51 us; skew 500 ppm
I20260812 11:15:21.019600 28275 webserver.cc:460] Webserver started at http://127.23.97.254:38865/ using document root <none> and password file <none>
I20260812 11:15:21.020226 28275 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.020292 28275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.020547 28275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.022660 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "02d348b9a74446849cee35a31f9a0e99"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "734727516b4828b92f2ea2da721d7e6c"
server_key_iv: "9f319ddc299fcd9878aeeac0aafbf8bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.023270 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "02d348b9a74446849cee35a31f9a0e99"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "734727516b4828b92f2ea2da721d7e6c"
server_key_iv: "9f319ddc299fcd9878aeeac0aafbf8bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.027177 28275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:21.029646 28296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.030826 28275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:21.030983 28275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "02d348b9a74446849cee35a31f9a0e99"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "734727516b4828b92f2ea2da721d7e6c"
server_key_iv: "9f319ddc299fcd9878aeeac0aafbf8bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.031100 28275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.043290 28275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.043931 28275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.044128 28275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.051407 28275 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:38719
I20260812 11:15:21.051433 28387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:38719 every 8 connection(s)
I20260812 11:15:21.052313 28275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:21.055467 28389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:21.056734 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28275
I20260812 11:15:21.056888 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:21.057206 23943 external_mini_cluster.cc:1428] Setting key 596d0d7b41620293050488f058375446
I20260812 11:15:21.061666 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: Bootstrap starting.
I20260812 11:15:21.064460 28389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:21.065269 28389 log.cc:824] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:21.066428 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.058552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.23.97.254:38719)
0812 11:15:21.059139 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.059154 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.059188 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:15:21.059815 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.059831 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.060200 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.060545 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.060887 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.061822 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.063020 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.063477 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.063590 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.064511 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.064517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.064529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:21.064954 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.065009 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:21.065031 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.065108 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.065288 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.065545 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260812 11:15:21.067289 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: No bootstrap required, opened a new log
I20260812 11:15:21.070860 28389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } }
I20260812 11:15:21.071107 28389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:21.071187 28389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d348b9a74446849cee35a31f9a0e99, State: Initialized, Role: FOLLOWER
I20260812 11:15:21.071600 28389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } }
I20260812 11:15:21.071784 28389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:21.071851 28389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:21.071966 28389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:21.072953 28389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } }
I20260812 11:15:21.073292 28389 leader_election.cc:302] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02d348b9a74446849cee35a31f9a0e99; no voters:
I20260812 11:15:21.073601 28389 leader_election.cc:288] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:21.073722 28395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:21.073948 28395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [term 1 LEADER]: Becoming Leader. State: Replica: 02d348b9a74446849cee35a31f9a0e99, State: Running, Role: LEADER
I20260812 11:15:21.074234 28395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } }
I20260812 11:15:21.074934 28389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:21.075366 28401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02d348b9a74446849cee35a31f9a0e99. Latest consensus state: current_term: 1 leader_uuid: "02d348b9a74446849cee35a31f9a0e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } } }
I20260812 11:15:21.075502 28401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:21.075807 28397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02d348b9a74446849cee35a31f9a0e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d348b9a74446849cee35a31f9a0e99" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 38719 } } }
I20260812 11:15:21.075914 28397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:21.075929 28403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:21.078929 28403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:21.084316 28403 catalog_manager.cc:1347] Generated new cluster ID: 0983ff7687ac4e3ea2ec269a61b1b18f
I20260812 11:15:21.084403 28403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:21.099376 28403 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:21.100688 28403 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:21.112787 28403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: Generated new TSK 0
I20260812 11:15:21.113632 28403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:21.121560 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.193:0
--local_ip_for_outbound_sockets=127.23.97.193
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--builtin_ntp_servers=127.23.97.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:15:21.236857 28435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:21.237120 28435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:21.237159 28435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:21.241233 28435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:21.241289 28435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:21.241343 28435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.193
I20260812 11:15:21.245922 28435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.97.193
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.193
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:21.246937 28435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.248416 28435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.258164 28444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.258164 28447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.258230 28449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.258601 28435 server_base.cc:1034] running on GCE node
I20260812 11:15:21.258992 28435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.259589 28435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.260751 28435 hybrid_clock.cc:648] HybridClock initialized: now 1786533321260740 us; error 26 us; skew 500 ppm
I20260812 11:15:21.262507 28435 webserver.cc:460] Webserver started at http://127.23.97.193:38369/ using document root <none> and password file <none>
I20260812 11:15:21.263027 28435 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.263087 28435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.263265 28435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.265349 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data/instance:
uuid: "005e223811c247b0b8171806bbd119c6"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "0c9eb64f1e15ed378e560eb0404fe985"
server_key_iv: "dac9c17260c44ac9902cfeeee89f51ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.265866 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance:
uuid: "005e223811c247b0b8171806bbd119c6"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "0c9eb64f1e15ed378e560eb0404fe985"
server_key_iv: "dac9c17260c44ac9902cfeeee89f51ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.269479 28435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:15:21.271864 28461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.273149 28435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:21.273267 28435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal
uuid: "005e223811c247b0b8171806bbd119c6"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "0c9eb64f1e15ed378e560eb0404fe985"
server_key_iv: "dac9c17260c44ac9902cfeeee89f51ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.273358 28435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.288079 28435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.288785 28435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.288937 28435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.289477 28435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:21.290354 28435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:21.290400 28435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.290441 28435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:21.290455 28435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.299204 28435 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.193:38173
I20260812 11:15:21.299300 28685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.193:38173 every 8 connection(s)
I20260812 11:15:21.300113 28435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/data/info.pb
I20260812 11:15:21.308950 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28435
I20260812 11:15:21.308998 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.301917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:42489 (local address 127.23.97.254:38719)
0812 11:15:21.302065 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.302070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.303130 (+ 1060us) server_negotiation.cc:407] Connection header received
0812 11:15:21.304512 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.304514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.304563 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.304640 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.305475 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.306050 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.307138 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.307359 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.307437 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.308478 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.308481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.308482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:21.308495 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.308511 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:21.308519 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.308578 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.308757 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.308829 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:15:21.309110 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-0/wal/instance
I20260812 11:15:21.309403 23943 external_mini_cluster.cc:1428] Setting key 26b49c65343fc71da47c249a6a65c3af
I20260812 11:15:21.309871 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.302345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38719 (local address 127.23.97.193:42489)
0812 11:15:21.302984 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.303022 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.304262 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.304806 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.304812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.304892 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.305346 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.305358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.306169 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.306172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.307031 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.307041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.307490 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.307492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.307624 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.308195 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:21.308197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.308234 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:21.308293 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.308593 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.308600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.308604 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.308713 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.308924 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260812 11:15:21.311097 28686 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38719
I20260812 11:15:21.311338 28686 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:21.311352 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.194:0
--local_ip_for_outbound_sockets=127.23.97.194
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--builtin_ntp_servers=127.23.97.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:21.311853 28686 heartbeater.cc:511] Master 127.23.97.254:38719 requested a full tablet report, sending...
I20260812 11:15:21.313499 28323 ts_manager.cc:194] Registered new tserver with Master: 005e223811c247b0b8171806bbd119c6 (127.23.97.193:38173)
I20260812 11:15:21.314653 28323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.193:42489
W20260812 11:15:21.428488 28694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:21.428804 28694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:21.428885 28694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:21.432989 28694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:21.433069 28694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:21.433162 28694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.194
I20260812 11:15:21.438273 28694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.97.194
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.194
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:21.439443 28694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.440976 28694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.450980 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.450969 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.450969 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.451334 28694 server_base.cc:1034] running on GCE node
I20260812 11:15:21.451689 28694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.452342 28694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.453598 28694 hybrid_clock.cc:648] HybridClock initialized: now 1786533321453582 us; error 27 us; skew 500 ppm
I20260812 11:15:21.455734 28694 webserver.cc:460] Webserver started at http://127.23.97.194:42303/ using document root <none> and password file <none>
I20260812 11:15:21.456414 28694 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.456493 28694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.456735 28694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.458820 28694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data/instance:
uuid: "1d9ee386327d47ea8c782ef6521efb19"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "bb2bfdd0dfd0322c39a5f65cfff3546a"
server_key_iv: "2a70238b28118264cad9f94ed7672e8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.459437 28694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance:
uuid: "1d9ee386327d47ea8c782ef6521efb19"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "bb2bfdd0dfd0322c39a5f65cfff3546a"
server_key_iv: "2a70238b28118264cad9f94ed7672e8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.463395 28694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:21.466041 28716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.467334 28694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:21.467482 28694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal
uuid: "1d9ee386327d47ea8c782ef6521efb19"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "bb2bfdd0dfd0322c39a5f65cfff3546a"
server_key_iv: "2a70238b28118264cad9f94ed7672e8b"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.467617 28694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.496889 28694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.497690 28694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.497920 28694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.498567 28694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:21.499567 28694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:21.499639 28694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.499711 28694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:21.499785 28694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.511162 28694 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.194:42945
I20260812 11:15:21.511188 28907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.194:42945 every 8 connection(s)
I20260812 11:15:21.512323 28694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/data/info.pb
I20260812 11:15:21.521575 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28694
I20260812 11:15:21.521731 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal/instance
I20260812 11:15:21.521878 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.514409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.194:46805 (local address 127.23.97.254:38719)
0812 11:15:21.514551 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.514554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.515560 (+ 1006us) server_negotiation.cc:407] Connection header received
0812 11:15:21.517053 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.517057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.517111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.517196 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.518123 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.518649 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.519897 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.520113 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.520198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.521303 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.521310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.521316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:21.521335 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.521361 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:21.521374 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.521459 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.521615 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.521697 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:15:21.522111 23943 external_mini_cluster.cc:1428] Setting key 9101d7faf5fa1806138fdc76d5d97e40
I20260812 11:15:21.522872 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.514771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38719 (local address 127.23.97.194:46805)
0812 11:15:21.515430 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.515471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.516829 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.517363 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.517369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.517453 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.517959 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.517972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.518760 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.518763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.519785 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.519795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.520243 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.520246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.520443 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.521016 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:21.521018 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.521059 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:21.521126 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.521466 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.521473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.521478 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.521574 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.521898 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260812 11:15:21.524132 28908 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38719
I20260812 11:15:21.524441 28908 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:21.524529 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.97.195:0
--local_ip_for_outbound_sockets=127.23.97.195
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--builtin_ntp_servers=127.23.97.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:15:21.525030 28908 heartbeater.cc:511] Master 127.23.97.254:38719 requested a full tablet report, sending...
I20260812 11:15:21.526147 28321 ts_manager.cc:194] Registered new tserver with Master: 1d9ee386327d47ea8c782ef6521efb19 (127.23.97.194:42945)
I20260812 11:15:21.526837 28321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.194:46805
W20260812 11:15:21.641232 28913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:21.641480 28913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:21.641517 28913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:21.645422 28913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:21.645475 28913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:21.645529 28913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.97.195
I20260812 11:15:21.650074 28913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.97.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.97.195
--webserver_port=0
--tserver_master_addrs=127.23.97.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.97.195
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:21.651033 28913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:21.652294 28913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:21.662575 28919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.662577 28924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:21.662577 28921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:21.664058 28913 server_base.cc:1034] running on GCE node
I20260812 11:15:21.664568 28913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:21.665264 28913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:21.666463 28913 hybrid_clock.cc:648] HybridClock initialized: now 1786533321666446 us; error 32 us; skew 500 ppm
I20260812 11:15:21.668292 28913 webserver.cc:460] Webserver started at http://127.23.97.195:40043/ using document root <none> and password file <none>
I20260812 11:15:21.668928 28913 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:21.669026 28913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:21.669298 28913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:21.671482 28913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data/instance:
uuid: "716f6a36b26245d0a35eb9d8b8105873"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "9932ba36a6f9141cf7c975dfab0725f5"
server_key_iv: "53a8871991a4ecee479877a7c28f3bb7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.672077 28913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance:
uuid: "716f6a36b26245d0a35eb9d8b8105873"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "9932ba36a6f9141cf7c975dfab0725f5"
server_key_iv: "53a8871991a4ecee479877a7c28f3bb7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.675783 28913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:15:21.678532 28935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.679719 28913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:21.679878 28913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal
uuid: "716f6a36b26245d0a35eb9d8b8105873"
format_stamp: "Formatted at 2026-08-12 11:15:21 on dist-test-slave-zpvx"
server_key: "9932ba36a6f9141cf7c975dfab0725f5"
server_key_iv: "53a8871991a4ecee479877a7c28f3bb7"
server_key_version: "encryptionkey@0"
I20260812 11:15:21.679989 28913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:21.692541 28913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:21.693186 28913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:21.693385 28913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:21.693962 28913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:15:21.694895 28913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:15:21.694967 28913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.695037 28913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:15:21.695078 28913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:21.704342 28913 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.195:43967
I20260812 11:15:21.704432 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.195:43967 every 8 connection(s)
I20260812 11:15:21.705307 28913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/data/info.pb
I20260812 11:15:21.712998 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 28913
I20260812 11:15:21.713158 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal/instance
I20260812 11:15:21.713472 23943 external_mini_cluster.cc:1428] Setting key b318901c8cd33e36dde35ff5812d0fdf
I20260812 11:15:21.715395 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.707287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.195:33497 (local address 127.23.97.254:38719)
0812 11:15:21.707435 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.707439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.708674 (+ 1235us) server_negotiation.cc:407] Connection header received
0812 11:15:21.710129 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.710133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.710193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.710284 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.711206 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.711753 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.713130 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.713427 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.713543 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.714811 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.714817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.714823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:21.714840 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.714864 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:21.714875 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.714954 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.715154 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.715215 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:15:21.716288 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.707714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:38719 (local address 127.23.97.195:33497)
0812 11:15:21.708480 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.708553 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.709907 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.710478 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.710484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.710569 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:21.711043 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.711055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.711909 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.711914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.712961 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.712974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.713594 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.713599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.713798 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.714543 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:15:21.714545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:21.714585 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:15:21.714658 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:21.714974 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:21.714984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:21.714990 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.715130 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.715354 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20260812 11:15:21.717505 29157 heartbeater.cc:347] Connected to a master server at 127.23.97.254:38719
I20260812 11:15:21.717757 29157 heartbeater.cc:464] Registering TS with master...
I20260812 11:15:21.718291 29157 heartbeater.cc:511] Master 127.23.97.254:38719 requested a full tablet report, sending...
I20260812 11:15:21.719311 28321 ts_manager.cc:194] Registered new tserver with Master: 716f6a36b26245d0a35eb9d8b8105873 (127.23.97.195:43967)
I20260812 11:15:21.719971 28321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.97.195:33497
I20260812 11:15:21.727607 23943 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:15:21.733688 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.729661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.23.97.254:38719)
0812 11:15:21.729829 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.729833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.730029 (+ 196us) server_negotiation.cc:407] Connection header received
0812 11:15:21.730206 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.730208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.730250 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.730330 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:21.730618 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.731172 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.732050 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.732268 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.732374 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.733256 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:21.733260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:21.733263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:21.733282 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:21.733298 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:21.733306 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:21.733385 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.733477 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.733533 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:15:21.736617 28321 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:15:21.739590 28321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:15:21.754375 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.748438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:38173 (local address 127.0.0.1:34092)
0812 11:15:21.749401 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.749416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.749521 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.750015 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.750019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.750037 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:21.750144 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.750150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751580 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.751586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.752966 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.752978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.753801 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.753911 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.753949 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.754067 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.754220 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":78,"thread_start_us":97,"threads_started":1}
I20260812 11:15:21.754567 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.748767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43967 (local address 127.0.0.1:55536)
0812 11:15:21.749697 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.749711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.749811 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.750224 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.750227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.750246 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:21.750347 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.750354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751301 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.751307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.752775 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.752785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.754059 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.754063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.754237 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.754256 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.754349 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.754422 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260812 11:15:21.754642 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.749221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.23.97.195:43967)
0812 11:15:21.749697 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.749714 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.749807 (+ 93us) server_negotiation.cc:407] Connection header received
0812 11:15:21.749914 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.749922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.750098 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.750250 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:21.750458 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751180 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.752901 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.753689 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753800 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.753994 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.754384 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.754482 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":161,"threads_started":1}
I20260812 11:15:21.755406 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.748025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42945 (local address 127.0.0.1:41028)
0812 11:15:21.748421 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.748471 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.748603 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.749801 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.749806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.749827 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:21.750009 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.750016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751638 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.751644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.752966 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.752978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.754075 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.754079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.755057 (+ 978us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.755090 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.755176 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.755258 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":891,"thread_start_us":90,"threads_started":1}
I20260812 11:15:21.755515 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.748175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.23.97.194:42945)
0812 11:15:21.748981 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.748995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.749022 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:15:21.749125 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.749132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.749442 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.749607 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:21.750146 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751527 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753101 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.753868 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753964 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.754122 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.755221 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.755336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":137,"threads_started":1}
I20260812 11:15:21.755761 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.748876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.23.97.193:38173)
0812 11:15:21.749378 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.749393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.749515 (+ 122us) server_negotiation.cc:407] Connection header received
0812 11:15:21.749637 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.749642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.749871 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.750026 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:21.750277 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.751442 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753147 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.753665 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.753795 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.753993 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.754221 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.754333 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":95,"threads_started":1}
I20260812 11:15:21.757741 29038 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa5f8c127d4f2286626645193d4efd (DEFAULT_TABLE table=test-table [id=b3a87fb7430e42f189d9f51f5c131bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:21.758020 28796 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa5f8c127d4f2286626645193d4efd (DEFAULT_TABLE table=test-table [id=b3a87fb7430e42f189d9f51f5c131bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:21.758914 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa5f8c127d4f2286626645193d4efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:21.759052 28796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa5f8c127d4f2286626645193d4efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:21.758836 28561 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa5f8c127d4f2286626645193d4efd (DEFAULT_TABLE table=test-table [id=b3a87fb7430e42f189d9f51f5c131bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:15:21.759739 28561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa5f8c127d4f2286626645193d4efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:21.765255 29186 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Bootstrap starting.
I20260812 11:15:21.766630 29188 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Bootstrap starting.
I20260812 11:15:21.767627 29186 tablet_bootstrap.cc:654] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:21.768353 29188 tablet_bootstrap.cc:654] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:21.768414 29186 log.cc:824] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:21.768718 29189 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Bootstrap starting.
I20260812 11:15:21.769281 29188 log.cc:824] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:21.770550 29186 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: No bootstrap required, opened a new log
I20260812 11:15:21.770830 29186 ts_tablet_manager.cc:1397] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 11:15:21.770960 29189 tablet_bootstrap.cc:654] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:21.771351 29188 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: No bootstrap required, opened a new log
I20260812 11:15:21.771533 29188 ts_tablet_manager.cc:1397] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 11:15:21.771816 29189 log.cc:824] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:21.773895 29189 tablet_bootstrap.cc:492] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: No bootstrap required, opened a new log
I20260812 11:15:21.774143 29189 ts_tablet_manager.cc:1397] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:15:21.774850 29186 raft_consensus.cc:348] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.775099 29186 raft_consensus.cc:374] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:21.775153 29186 raft_consensus.cc:729] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ee386327d47ea8c782ef6521efb19, State: Initialized, Role: FOLLOWER
I20260812 11:15:21.775696 29186 consensus_queue.cc:260] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.775990 29188 raft_consensus.cc:348] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.776247 29188 raft_consensus.cc:374] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:21.776348 29188 raft_consensus.cc:729] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716f6a36b26245d0a35eb9d8b8105873, State: Initialized, Role: FOLLOWER
I20260812 11:15:21.776839 29188 consensus_queue.cc:260] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.776932 29186 ts_tablet_manager.cc:1428] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 11:15:21.777436 28908 heartbeater.cc:503] Master 127.23.97.254:38719 was elected leader, sending a full tablet report...
I20260812 11:15:21.777736 29157 heartbeater.cc:503] Master 127.23.97.254:38719 was elected leader, sending a full tablet report...
I20260812 11:15:21.777835 29188 ts_tablet_manager.cc:1428] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 11:15:21.778854 29189 raft_consensus.cc:348] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.779098 29189 raft_consensus.cc:374] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:21.779155 29189 raft_consensus.cc:729] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005e223811c247b0b8171806bbd119c6, State: Initialized, Role: FOLLOWER
I20260812 11:15:21.779546 29189 consensus_queue.cc:260] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.780244 28686 heartbeater.cc:503] Master 127.23.97.254:38719 was elected leader, sending a full tablet report...
I20260812 11:15:21.780678 29189 ts_tablet_manager.cc:1428] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260812 11:15:21.801259 28688 tablet.cc:2369] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:15:21.809898 29198 raft_consensus.cc:482] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:15:21.810093 29198 raft_consensus.cc:504] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.811285 29198 leader_election.cc:288] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9ee386327d47ea8c782ef6521efb19 (127.23.97.194:42945), 716f6a36b26245d0a35eb9d8b8105873 (127.23.97.195:43967)
I20260812 11:15:21.815157 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.811686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:41269 (local address 127.23.97.194:42945)
0812 11:15:21.811837 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.811842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.811942 (+ 100us) server_negotiation.cc:407] Connection header received
0812 11:15:21.812125 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.812128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.812189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.812331 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:21.812652 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.813152 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814218 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.814612 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814739 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.814791 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.814971 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.815034 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:15:21.815178 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.811531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42945 (local address 127.23.97.193:41269)
0812 11:15:21.811843 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.811860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.812000 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.812344 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.812346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.812395 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:21.812536 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.812541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.813273 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.813279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.814082 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.814091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.814743 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.814837 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.814860 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.814954 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.815021 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260812 11:15:21.815402 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.812086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.97.193:50855 (local address 127.23.97.195:43967)
0812 11:15:21.812229 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.812261 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.812359 (+ 98us) server_negotiation.cc:407] Connection header received
0812 11:15:21.812532 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.812536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.812585 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.812668 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:15:21.812949 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.813439 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814468 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.814853 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814988 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.815032 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.815224 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.815293 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:15:21.815421 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.811960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.195:43967 (local address 127.23.97.193:50855)
0812 11:15:21.812260 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.812274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:21.812418 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:21.812681 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:21.812684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:21.812701 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:15:21.812840 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.812846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.813553 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.813557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.814344 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:21.814353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.814988 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.814991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:21.815090 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.815120 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:15:21.815209 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:15:21.815276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260812 11:15:21.815881 28840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa5f8c127d4f2286626645193d4efd" candidate_uuid: "005e223811c247b0b8171806bbd119c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ee386327d47ea8c782ef6521efb19" is_pre_election: true
I20260812 11:15:21.815963 29073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa5f8c127d4f2286626645193d4efd" candidate_uuid: "005e223811c247b0b8171806bbd119c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716f6a36b26245d0a35eb9d8b8105873" is_pre_election: true
I20260812 11:15:21.816193 28840 raft_consensus.cc:2454] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 005e223811c247b0b8171806bbd119c6 in term 0.
I20260812 11:15:21.816250 29073 raft_consensus.cc:2454] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 005e223811c247b0b8171806bbd119c6 in term 0.
I20260812 11:15:21.816720 28465 leader_election.cc:302] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005e223811c247b0b8171806bbd119c6, 1d9ee386327d47ea8c782ef6521efb19; no voters:
I20260812 11:15:21.817063 29198 raft_consensus.cc:2781] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:15:21.817140 29198 raft_consensus.cc:482] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:15:21.817171 29198 raft_consensus.cc:3037] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:21.818289 29198 raft_consensus.cc:504] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.818645 29198 leader_election.cc:288] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9ee386327d47ea8c782ef6521efb19 (127.23.97.194:42945), 716f6a36b26245d0a35eb9d8b8105873 (127.23.97.195:43967)
I20260812 11:15:21.819118 28840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa5f8c127d4f2286626645193d4efd" candidate_uuid: "005e223811c247b0b8171806bbd119c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ee386327d47ea8c782ef6521efb19"
I20260812 11:15:21.819124 29073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa5f8c127d4f2286626645193d4efd" candidate_uuid: "005e223811c247b0b8171806bbd119c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716f6a36b26245d0a35eb9d8b8105873"
I20260812 11:15:21.819267 28840 raft_consensus.cc:3037] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:21.819255 29073 raft_consensus.cc:3037] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:21.820530 29073 raft_consensus.cc:2454] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 005e223811c247b0b8171806bbd119c6 in term 1.
I20260812 11:15:21.820657 28840 raft_consensus.cc:2454] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 005e223811c247b0b8171806bbd119c6 in term 1.
I20260812 11:15:21.820896 28465 leader_election.cc:302] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005e223811c247b0b8171806bbd119c6, 716f6a36b26245d0a35eb9d8b8105873; no voters:
I20260812 11:15:21.821094 29198 raft_consensus.cc:2781] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:21.821331 29198 raft_consensus.cc:686] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 1 LEADER]: Becoming Leader. State: Replica: 005e223811c247b0b8171806bbd119c6, State: Running, Role: LEADER
I20260812 11:15:21.821687 29198 consensus_queue.cc:237] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } }
I20260812 11:15:21.824954 28323 catalog_manager.cc:5495] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 005e223811c247b0b8171806bbd119c6 (127.23.97.193). New cstate: current_term: 1 leader_uuid: "005e223811c247b0b8171806bbd119c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "005e223811c247b0b8171806bbd119c6" member_type: VOTER last_known_addr { host: "127.23.97.193" port: 38173 } health_report { overall_health: HEALTHY } } }
I20260812 11:15:21.851610 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.848325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.23.97.193:38173)
0812 11:15:21.848470 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:21.848474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:21.848489 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:15:21.848629 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:21.848631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:21.848692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:21.848787 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:21.849009 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.849509 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.850335 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:21.850548 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:21.850637 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:21.850896 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:21.851305 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:21.851395 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:21.851456 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 11:15:21.851502 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:21.857517 29073 raft_consensus.cc:1261] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 1 FOLLOWER]: Refusing update from remote peer 005e223811c247b0b8171806bbd119c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:21.857532 28840 raft_consensus.cc:1261] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 1 FOLLOWER]: Refusing update from remote peer 005e223811c247b0b8171806bbd119c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:21.858279 29198 consensus_queue.cc:1035] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "716f6a36b26245d0a35eb9d8b8105873" member_type: VOTER last_known_addr { host: "127.23.97.195" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:21.858464 29204 consensus_queue.cc:1035] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9ee386327d47ea8c782ef6521efb19" member_type: VOTER last_known_addr { host: "127.23.97.194" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:21.865418 29211 mvcc.cc:205] Tried to move back new op lower bound from 7317640486321029120 to 7317640486183944192. Current Snapshot: MvccSnapshot[applied={T|T < 7317640486321029120}]
I20260812 11:15:21.867633 29215 mvcc.cc:205] Tried to move back new op lower bound from 7317640486321029120 to 7317640486183944192. Current Snapshot: MvccSnapshot[applied={T|T < 7317640486321029120}]
I20260812 11:15:21.868163 29213 mvcc.cc:205] Tried to move back new op lower bound from 7317640486321029120 to 7317640486183944192. Current Snapshot: MvccSnapshot[applied={T|T < 7317640486321029120}]
I20260812 11:15:21.875320 28321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:21.875507 28321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:15:21.878091 28321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02d348b9a74446849cee35a31f9a0e99: Sending DeleteTablet for 3 replicas of tablet 5daa5f8c127d4f2286626645193d4efd
I20260812 11:15:21.878971 28561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa5f8c127d4f2286626645193d4efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:21 UTC) from {username='slave'} at 127.0.0.1:34092
I20260812 11:15:21.879094 29038 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa5f8c127d4f2286626645193d4efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:21 UTC) from {username='slave'} at 127.0.0.1:55536
I20260812 11:15:21.879492 29228 tablet_replica.cc:330] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6: stopping tablet replica
I20260812 11:15:21.879492 28796 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa5f8c127d4f2286626645193d4efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:15:21 UTC) from {username='slave'} at 127.0.0.1:41028
I20260812 11:15:21.879813 29228 raft_consensus.cc:2229] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:15:21.879993 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28435
I20260812 11:15:21.880239 29228 raft_consensus.cc:2258] T 5daa5f8c127d4f2286626645193d4efd P 005e223811c247b0b8171806bbd119c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:21.892743 29232 tablet_replica.cc:330] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: stopping tablet replica
I20260812 11:15:21.892745 29230 tablet_replica.cc:330] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: stopping tablet replica
I20260812 11:15:21.893196 29232 raft_consensus.cc:2229] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:21.893213 29230 raft_consensus.cc:2229] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:15:21.893573 29232 raft_consensus.cc:2258] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:21.893579 29230 raft_consensus.cc:2258] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:15:21.895263 29232 ts_tablet_manager.cc:1905] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:21.895275 29230 ts_tablet_manager.cc:1905] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:15:21.897907 29232 ts_tablet_manager.cc:1918] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:21.898029 29232 log.cc:1196] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-1/wal/wals/5daa5f8c127d4f2286626645193d4efd
I20260812 11:15:21.898384 29232 ts_tablet_manager.cc:1939] T 5daa5f8c127d4f2286626645193d4efd P 1d9ee386327d47ea8c782ef6521efb19: Deleting consensus metadata
I20260812 11:15:21.899335 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28694
W20260812 11:15:21.899299 28301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.97.193:38173 (error 108)
I20260812 11:15:21.899313 29230 ts_tablet_manager.cc:1918] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:15:21.899523 29230 log.cc:1196] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Deleting WAL directory at /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533060973827-23943-0/minicluster-data/ts-2/wal/wals/5daa5f8c127d4f2286626645193d4efd
I20260812 11:15:21.900084 29230 ts_tablet_manager.cc:1939] T 5daa5f8c127d4f2286626645193d4efd P 716f6a36b26245d0a35eb9d8b8105873: Deleting consensus metadata
I20260812 11:15:21.901697 28297 catalog_manager.cc:4841] TS 716f6a36b26245d0a35eb9d8b8105873 (127.23.97.195:43967): tablet 5daa5f8c127d4f2286626645193d4efd (table test-table [id=b3a87fb7430e42f189d9f51f5c131bc5]) successfully deleted
I20260812 11:15:21.909451 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.906597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.193:38173 (local address 127.0.0.1:34106)
0812 11:15:21.909240 (+ 2643us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.909340 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.193:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2526}
W20260812 11:15:21.923130 28301 catalog_manager.cc:4584] TS 005e223811c247b0b8171806bbd119c6 (127.23.97.193:38173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5daa5f8c127d4f2286626645193d4efd: Network error: Client connection negotiation failed: client connection to 127.23.97.193:38173: connect: Connection refused (error 111)
I20260812 11:15:21.929646 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28913
I20260812 11:15:21.933367 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:21.929956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.194:42945 (local address 127.0.0.1:41032)
0812 11:15:21.933155 (+ 3199us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:21.933256 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.194:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3072}
I20260812 11:15:21.947371 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 28275
2026-08-12T11:15:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1112 ms)
[----------] 6 tests from EncryptionPolicyTest (11332 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:22 dist-test-slave-zpvx krb5kdc[29239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:22 dist-test-slave-zpvx krb5kdc[29239](info): set up 2 sockets
Aug 12 11:15:22 dist-test-slave-zpvx krb5kdc[29239](info): commencing operation
krb5kdc: starting...
W20260812 11:15:23.291697 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.308s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:23.427199 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:23.550261 29265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:23.550595 29265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:23.550684 29265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:23.554992 29265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:23.555087 29265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:23.555125 29265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:23.555164 29265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:23.555209 29265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:23.560961 29265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:23.562176 29265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:23.563354 29265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:23.572417 29282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:23.572391 29278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:23.572517 29265 server_base.cc:1034] running on GCE node
W20260812 11:15:23.572422 29277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:23.573378 29265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:23.574548 29265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:23.575737 29265 hybrid_clock.cc:648] HybridClock initialized: now 1786533323575725 us; error 29 us; skew 500 ppm
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:23.583099 29265 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:23.584192 29265 webserver.cc:460] Webserver started at http://127.0.0.1:46359/ using document root <none> and password file <none>
I20260812 11:15:23.584916 29265 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:23.585024 29265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:23.585289 29265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:23.587544 29265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "c35422517bef44e5afb1b4e48691c6af"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-zpvx"
server_key: "47d3e1c483df5086d4bb8e58dec0b035"
server_key_iv: "381d37ea6b8d69c106825d14f18acc51"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.588189 29265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "c35422517bef44e5afb1b4e48691c6af"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-zpvx"
server_key: "47d3e1c483df5086d4bb8e58dec0b035"
server_key_iv: "381d37ea6b8d69c106825d14f18acc51"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.592319 29265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:15:23.595112 29296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:23.596714 29265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:23.596887 29265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c35422517bef44e5afb1b4e48691c6af"
format_stamp: "Formatted at 2026-08-12 11:15:23 on dist-test-slave-zpvx"
server_key: "47d3e1c483df5086d4bb8e58dec0b035"
server_key_iv: "381d37ea6b8d69c106825d14f18acc51"
server_key_version: "encryptionkey@0"
I20260812 11:15:23.597035 29265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:23.632850 29265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:23.635982 29265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:23.636231 29265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:23.644230 29265 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36309
I20260812 11:15:23.644276 29376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36309 every 8 connection(s)
I20260812 11:15:23.645495 29265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:23.645816 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29265
I20260812 11:15:23.645989 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:23.646386 23943 external_mini_cluster.cc:1428] Setting key 6df9cbeea9f57aacfe91a472f4ea9a1f
I20260812 11:15:23.649245 29377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:23.656201 29377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af: Bootstrap starting.
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:15:23.659058 29377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:23.660019 29377 log.cc:824] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:23.662315 29377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af: No bootstrap required, opened a new log
I20260812 11:15:23.662510 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:23.648077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45551 (local address 127.0.0.1:36309)
0812 11:15:23.648555 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:23.648576 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:23.648623 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:15:23.649535 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:23.649559 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:23.650008 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:23.650405 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:23.651164 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.652742 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.653644 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.654339 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.654514 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:23.657418 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:23.657423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:23.657437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:23.657466 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:23.659307 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.659918 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.659924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.659929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.660046 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.660470 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.660473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.660474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.660894 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:23.661068 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:23.661435 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 11:15:23.661695 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":120,"threads_started":1}
I20260812 11:15:23.666276 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:23.666333 29377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } }
I20260812 11:15:23.666633 29377 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:23.666694 29377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35422517bef44e5afb1b4e48691c6af, State: Initialized, Role: FOLLOWER
I20260812 11:15:23.667135 29377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } }
I20260812 11:15:23.667323 29377 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:23.667429 29377 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:23.667544 29377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:23.668984 29377 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } }
I20260812 11:15:23.669407 29377 leader_election.cc:302] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35422517bef44e5afb1b4e48691c6af; no voters:
I20260812 11:15:23.669821 29377 leader_election.cc:288] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:23.670230 29400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:23.670459 29400 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [term 1 LEADER]: Becoming Leader. State: Replica: c35422517bef44e5afb1b4e48691c6af, State: Running, Role: LEADER
I20260812 11:15:23.670807 29400 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } }
I20260812 11:15:23.671275 29377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:23.672072 29399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35422517bef44e5afb1b4e48691c6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } } }
I20260812 11:15:23.672078 29400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35422517bef44e5afb1b4e48691c6af. Latest consensus state: current_term: 1 leader_uuid: "c35422517bef44e5afb1b4e48691c6af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35422517bef44e5afb1b4e48691c6af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36309 } } }
I20260812 11:15:23.672390 29399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:23.672420 29400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:23.673146 29419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:23.676829 29419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:23.679018 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:23.668808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57589 (local address 127.0.0.1:36309)
0812 11:15:23.668961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:23.668966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:23.669237 (+ 271us) server_negotiation.cc:407] Connection header received
0812 11:15:23.669446 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:23.669449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:23.669527 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:23.669587 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:23.670703 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.671502 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.672740 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:23.673121 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:23.673243 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:23.675288 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:23.675291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:23.675293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:23.675316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:23.676967 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.677491 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.677494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.677496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.677587 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:23.677935 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:23.677938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:23.677940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:23.678134 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:23.678300 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:23.678580 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 11:15:23.678679 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260812 11:15:23.683076 29419 catalog_manager.cc:1347] Generated new cluster ID: 373c4155e39244f18c9492b6aa942a51
I20260812 11:15:23.683145 29419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:23.700523 29419 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:23.701593 29419 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:23.709398 29419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35422517bef44e5afb1b4e48691c6af: Generated new TSK 0
I20260812 11:15:23.710304 29419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:23.723347 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29265
W20260812 11:15:23.737352 29237 connection.cc:535] client connection to 127.0.0.1:36309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36309): Connection reset by peer (error 104)
2026-08-12T11:15:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29435](info): set up 2 sockets
Aug 12 11:15:23 dist-test-slave-zpvx krb5kdc[29435](info): commencing operation
krb5kdc: starting...
W20260812 11:15:25.041747 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:25 dist-test-slave-zpvx krb5kdc[29435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:25.171976 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:25.285138 29455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:25.285436 29455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:25.285507 29455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:25.289513 29455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:25.289597 29455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:25.289635 29455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:25.289690 29455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:25.289721 29455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:25.294823 29455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:25.295854 29455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:25.296939 29455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:25.305750 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.305920 29455 server_base.cc:1034] running on GCE node
W20260812 11:15:25.305750 29468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:25.305792 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:25.306770 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:25.307797 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:25.308962 29455 hybrid_clock.cc:648] HybridClock initialized: now 1786533325308943 us; error 30 us; skew 500 ppm
Aug 12 11:15:25 dist-test-slave-zpvx krb5kdc[29435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:25.311756 29455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:25.312649 29455 webserver.cc:460] Webserver started at http://127.0.0.1:38545/ using document root <none> and password file <none>
I20260812 11:15:25.313161 29455 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:25.313249 29455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:25.313426 29455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:25.315548 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "872bb9a4aaaa49c682da53ad456ddab5"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-zpvx"
server_key: "fd4623bd90a4dda2bf8a8a0792c976c9"
server_key_iv: "400d8d5fe918cc075a8c41140ccf906b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.316167 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "872bb9a4aaaa49c682da53ad456ddab5"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-zpvx"
server_key: "fd4623bd90a4dda2bf8a8a0792c976c9"
server_key_iv: "400d8d5fe918cc075a8c41140ccf906b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.319974 29455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:15:25.322551 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:25.323915 29455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:25.324031 29455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "872bb9a4aaaa49c682da53ad456ddab5"
format_stamp: "Formatted at 2026-08-12 11:15:25 on dist-test-slave-zpvx"
server_key: "fd4623bd90a4dda2bf8a8a0792c976c9"
server_key_iv: "400d8d5fe918cc075a8c41140ccf906b"
server_key_version: "encryptionkey@0"
I20260812 11:15:25.324160 29455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:25.350945 29455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:25.354380 29455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:25.354596 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:25.361578 29455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33335
I20260812 11:15:25.361644 29578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33335 every 8 connection(s)
I20260812 11:15:25.362571 29455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:25.365520 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:25.369625 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29455
I20260812 11:15:25.369773 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:25.370071 23943 external_mini_cluster.cc:1428] Setting key d76c0997ba8ef78895a0a02db8e35ce3
I20260812 11:15:25.371902 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5: Bootstrap starting.
I20260812 11:15:25.374337 29580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:25.375201 29580 log.cc:824] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:25.377391 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5: No bootstrap required, opened a new log
I20260812 11:15:25.379364 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.371538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54741 (local address 127.0.0.1:33335)
0812 11:15:25.371954 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.371969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.372005 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:15:25.372718 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.372736 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.373068 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.373339 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.373734 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.374750 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.375920 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.376447 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.376567 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.377506 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.377513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.377535 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:25.377908 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.377962 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:25.377984 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.378085 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.378302 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.378455 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260812 11:15:25.380511 29580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 11:15:25.380720 29580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:25.380792 29580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872bb9a4aaaa49c682da53ad456ddab5, State: Initialized, Role: FOLLOWER
I20260812 11:15:25.381295 29580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 11:15:25.381484 29580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:25.381562 29580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:25.381668 29580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:25.382294 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:25.382681 29580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 11:15:25.383021 29580 leader_election.cc:302] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 872bb9a4aaaa49c682da53ad456ddab5; no voters:
I20260812 11:15:25.383386 29580 leader_election.cc:288] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:25.383448 29595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:25.384792 29580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:25.384825 29595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [term 1 LEADER]: Becoming Leader. State: Replica: 872bb9a4aaaa49c682da53ad456ddab5, State: Running, Role: LEADER
I20260812 11:15:25.385138 29595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260812 11:15:25.386475 29595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "872bb9a4aaaa49c682da53ad456ddab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260812 11:15:25.386673 29595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:25.386909 29607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 872bb9a4aaaa49c682da53ad456ddab5. Latest consensus state: current_term: 1 leader_uuid: "872bb9a4aaaa49c682da53ad456ddab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872bb9a4aaaa49c682da53ad456ddab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260812 11:15:25.387019 29607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:25.387408 29631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:25.389350 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:25.384663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57659 (local address 127.0.0.1:33335)
0812 11:15:25.384801 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:25.384808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:25.384822 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:15:25.384998 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:25.385001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:25.385038 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:25.385130 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:25.385428 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.386152 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.387438 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:25.387711 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:25.387790 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:25.388799 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:25.388802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:25.388805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:25.388818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:25.388844 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:25.388872 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:25.388968 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:25.389192 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 11:15:25.389229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:15:25.390928 29631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:25.402532 29631 catalog_manager.cc:1347] Generated new cluster ID: 5310210a41c449b69b911dfb4117e6d9
I20260812 11:15:25.402654 29631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:25.412402 29631 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:25.413296 29631 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:25.418296 29631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 872bb9a4aaaa49c682da53ad456ddab5: Generated new TSK 0
I20260812 11:15:25.418938 29631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:25.426117 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29455
W20260812 11:15:25.443264 29432 connection.cc:535] client connection to 127.0.0.1:33335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33335): Connection reset by peer (error 104)
2026-08-12T11:15:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1705 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:25 dist-test-slave-zpvx krb5kdc[29645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:25 dist-test-slave-zpvx krb5kdc[29645](info): set up 2 sockets
Aug 12 11:15:25 dist-test-slave-zpvx krb5kdc[29645](info): commencing operation
krb5kdc: starting...
W20260812 11:15:26.750068 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:26 dist-test-slave-zpvx krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:26.879211 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:26.991624 29665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:26.991925 29665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:26.991997 29665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:26.995905 29665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:26.995994 29665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:26.996034 29665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:26.996086 29665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:26.996122 29665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:27.001210 29665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:27.002209 29665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:27.003199 29665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:27.011724 29680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:27.011790 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:27.011749 29678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:27.012018 29665 server_base.cc:1034] running on GCE node
I20260812 11:15:27.012682 29665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:27.013761 29665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:27.014917 29665 hybrid_clock.cc:648] HybridClock initialized: now 1786533327014889 us; error 42 us; skew 500 ppm
Aug 12 11:15:27 dist-test-slave-zpvx krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:27.017786 29665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:27.018673 29665 webserver.cc:460] Webserver started at http://127.0.0.1:35139/ using document root <none> and password file <none>
I20260812 11:15:27.019222 29665 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:27.019320 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:27.019541 29665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:27.021587 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "1a2574c2bf4845f48f02d4c409ab43b3"
format_stamp: "Formatted at 2026-08-12 11:15:27 on dist-test-slave-zpvx"
server_key: "477e70f1a7aeefaf6e64f640979b435d"
server_key_iv: "d0c753d5dea93e123799d230e4700cdb"
server_key_version: "encryptionkey@0"
I20260812 11:15:27.022142 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "1a2574c2bf4845f48f02d4c409ab43b3"
format_stamp: "Formatted at 2026-08-12 11:15:27 on dist-test-slave-zpvx"
server_key: "477e70f1a7aeefaf6e64f640979b435d"
server_key_iv: "d0c753d5dea93e123799d230e4700cdb"
server_key_version: "encryptionkey@0"
I20260812 11:15:27.025871 29665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:15:27.028203 29690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:27.029604 29665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:15:27.029726 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "1a2574c2bf4845f48f02d4c409ab43b3"
format_stamp: "Formatted at 2026-08-12 11:15:27 on dist-test-slave-zpvx"
server_key: "477e70f1a7aeefaf6e64f640979b435d"
server_key_iv: "d0c753d5dea93e123799d230e4700cdb"
server_key_version: "encryptionkey@0"
I20260812 11:15:27.029810 29665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:27.034147 29665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:27.037302 29665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:27.037443 29665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:27.043972 29665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45817
I20260812 11:15:27.043988 29773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45817 every 8 connection(s)
I20260812 11:15:27.044997 29665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:27.045409 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29665
I20260812 11:15:27.045568 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:27.045907 23943 external_mini_cluster.cc:1428] Setting key 6d545adb8d84c585444edc6abdb16977
I20260812 11:15:27.048112 29774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:27.051430 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.047292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41723 (local address 127.0.0.1:45817)
0812 11:15:27.047696 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.047715 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.047758 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:15:27.048590 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.048613 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.048957 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.049337 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.050059 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.050065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.050078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:27.050091 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.050151 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:27.050176 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.050329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.050412 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.050562 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260812 11:15:27.054265 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:27.054844 29774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3: Bootstrap starting.
I20260812 11:15:27.057220 29774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:27.057873 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:27.056253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49385 (local address 127.0.0.1:45817)
0812 11:15:27.056365 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:27.056385 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:27.056566 (+ 181us) server_negotiation.cc:407] Connection header received
0812 11:15:27.056709 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:27.056712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:27.056745 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:27.056823 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:27.057505 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:27.057509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:27.057511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:27.057522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:27.057552 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:27.057563 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:27.057648 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:27.057725 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 11:15:27.057763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 11:15:27.058041 29774 log.cc:824] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:27.059835 29774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3: No bootstrap required, opened a new log
I20260812 11:15:27.063382 29774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260812 11:15:27.063585 29774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:27.063628 29774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2574c2bf4845f48f02d4c409ab43b3, State: Initialized, Role: FOLLOWER
I20260812 11:15:27.064064 29774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260812 11:15:27.064215 29774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:27.064325 29774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:27.064455 29774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:27.065281 29774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260812 11:15:27.065608 29774 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2574c2bf4845f48f02d4c409ab43b3; no voters:
I20260812 11:15:27.065894 29774 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:27.066035 29789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:27.066272 29789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2574c2bf4845f48f02d4c409ab43b3, State: Running, Role: LEADER
I20260812 11:15:27.066610 29789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260812 11:15:27.067097 29774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:27.068357 29791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2574c2bf4845f48f02d4c409ab43b3. Latest consensus state: current_term: 1 leader_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260812 11:15:27.068359 29790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2574c2bf4845f48f02d4c409ab43b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260812 11:15:27.068570 29791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.068588 29790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:27.068977 29804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:27.071681 29804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:27.076663 29804 catalog_manager.cc:1347] Generated new cluster ID: afce39e21bd54c30bb707f257252fb11
I20260812 11:15:27.076736 29804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:27.113940 29804 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:27.114321 29804 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:27.122409 29804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2574c2bf4845f48f02d4c409ab43b3: Generated new TSK 0
I20260812 11:15:27.123234 29804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:27.138816 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29665
2026-08-12T11:15:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1712 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:27 dist-test-slave-zpvx krb5kdc[29822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:27 dist-test-slave-zpvx krb5kdc[29822](info): set up 2 sockets
Aug 12 11:15:27 dist-test-slave-zpvx krb5kdc[29822](info): commencing operation
krb5kdc: starting...
W20260812 11:15:28.470027 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:28 dist-test-slave-zpvx krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:28.594966 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:28.707908 29846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:28.708186 29846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:28.708289 29846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:28.712209 29846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:28.712292 29846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:28.712332 29846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:28.712396 29846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:28.712431 29846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:28.717417 29846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:28.718369 29846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:28.719386 29846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:28.728135 29860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:28.728202 29855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:28.728355 29846 server_base.cc:1034] running on GCE node
W20260812 11:15:28.728155 29858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:28.729084 29846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:28.730239 29846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:28.731441 29846 hybrid_clock.cc:648] HybridClock initialized: now 1786533328731410 us; error 42 us; skew 500 ppm
W20260812 11:15:29.642189 29846 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-zpvx': real 0.910s user 0.000s sys 0.001s
Aug 12 11:15:29 dist-test-slave-zpvx krb5kdc[29822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:29.644448 29846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:29.645551 29846 webserver.cc:460] Webserver started at http://127.0.0.1:38643/ using document root <none> and password file <none>
I20260812 11:15:29.646152 29846 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:29.646252 29846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:29.646495 29846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:29.648784 29846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "bcbff2a45d3442f28ebf6554e0cfe105"
format_stamp: "Formatted at 2026-08-12 11:15:29 on dist-test-slave-zpvx"
server_key: "1f075add0e7126d0333fbc9427c213e9"
server_key_iv: "602539f5388fb552aeaf3d5b56dd8402"
server_key_version: "encryptionkey@0"
I20260812 11:15:29.649394 29846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "bcbff2a45d3442f28ebf6554e0cfe105"
format_stamp: "Formatted at 2026-08-12 11:15:29 on dist-test-slave-zpvx"
server_key: "1f075add0e7126d0333fbc9427c213e9"
server_key_iv: "602539f5388fb552aeaf3d5b56dd8402"
server_key_version: "encryptionkey@0"
I20260812 11:15:29.653299 29846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:15:29.655809 29871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:29.657063 29846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:15:29.657207 29846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "bcbff2a45d3442f28ebf6554e0cfe105"
format_stamp: "Formatted at 2026-08-12 11:15:29 on dist-test-slave-zpvx"
server_key: "1f075add0e7126d0333fbc9427c213e9"
server_key_iv: "602539f5388fb552aeaf3d5b56dd8402"
server_key_version: "encryptionkey@0"
I20260812 11:15:29.657330 29846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:29.674535 29846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:29.677506 29846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:29.677697 29846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:29.684847 29846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38349
I20260812 11:15:29.684872 29968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38349 every 8 connection(s)
I20260812 11:15:29.685765 29846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:29.688416 29971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:29.694367 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 29846
I20260812 11:15:29.694507 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:29.694787 23943 external_mini_cluster.cc:1428] Setting key 352d70f7245b0cfa191596be0de839c3
I20260812 11:15:29.694994 29971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105: Bootstrap starting.
I20260812 11:15:29.697530 29971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:29.698276 29971 log.cc:824] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:29.700112 29971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105: No bootstrap required, opened a new log
Aug 12 11:15:29 dist-test-slave-zpvx krb5kdc[29822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:15:29.704124 29971 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } }
I20260812 11:15:29.704417 29971 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:29.704488 29971 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbff2a45d3442f28ebf6554e0cfe105, State: Initialized, Role: FOLLOWER
I20260812 11:15:29.704983 29971 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } }
I20260812 11:15:29.705111 29971 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:29.705194 29971 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:29.705303 29971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:29.706424 29971 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } }
I20260812 11:15:29.706816 29971 leader_election.cc:302] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbff2a45d3442f28ebf6554e0cfe105; no voters:
I20260812 11:15:29.707157 29971 leader_election.cc:288] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:29.707253 29981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:29.707547 29981 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [term 1 LEADER]: Becoming Leader. State: Replica: bcbff2a45d3442f28ebf6554e0cfe105, State: Running, Role: LEADER
I20260812 11:15:29.707885 29981 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } }
I20260812 11:15:29.708487 29971 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:29.709198 29981 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbff2a45d3442f28ebf6554e0cfe105. Latest consensus state: current_term: 1 leader_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } } }
I20260812 11:15:29.709329 29981 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:29.709652 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.696109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42009 (local address 127.0.0.1:38349)
0812 11:15:29.696595 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:29.696610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:29.696645 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:29.697278 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:29.697295 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:29.697623 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:29.698001 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:29.698823 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.699794 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.700753 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.701224 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.701402 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.704152 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:29.704158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:29.704175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:29.704206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:29.706414 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:29.706869 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:29.706875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:29.706880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:29.707025 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:29.707352 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:29.707358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:29.707360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:29.707839 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:29.708035 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:29.708334 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 11:15:29.708675 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":153,"threads_started":1}
I20260812 11:15:29.709640 29982 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbff2a45d3442f28ebf6554e0cfe105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38349 } } }
I20260812 11:15:29.709947 29982 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:29.711444 29989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:29.712832 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:29.714514 29989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:29.721068 29989 catalog_manager.cc:1347] Generated new cluster ID: 63dc182ed51e46549db91148e9101ced
I20260812 11:15:29.721453 29989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:29.723551 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:29.714957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47017 (local address 127.0.0.1:38349)
0812 11:15:29.715060 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:29.715066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:29.715392 (+ 326us) server_negotiation.cc:407] Connection header received
0812 11:15:29.715530 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:29.715534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:29.715583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:29.715662 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:29.716438 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.717086 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.718246 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:29.718554 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:29.718675 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:29.720212 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:29.720216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:29.720218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:29.720242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:29.721917 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:29.722340 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:29.722343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:29.722345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:29.722424 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:29.722704 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:29.722708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:29.722709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:29.722876 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:29.723006 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:29.723264 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 11:15:29.723378 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 11:15:29.732776 29989 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:29.733888 29989 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:29.744758 29989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbff2a45d3442f28ebf6554e0cfe105: Generated new TSK 0
I20260812 11:15:29.745450 29989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:29.749441 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 29846
2026-08-12T11:15:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:29 dist-test-slave-zpvx krb5kdc[30023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:29 dist-test-slave-zpvx krb5kdc[30023](info): set up 2 sockets
Aug 12 11:15:29 dist-test-slave-zpvx krb5kdc[30023](info): commencing operation
krb5kdc: starting...
W20260812 11:15:31.070544 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:31 dist-test-slave-zpvx krb5kdc[30023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:31.199142 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:31.310586 30051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:31.310827 30051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:31.310868 30051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:31.314828 30051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:31.314886 30051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:31.314905 30051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:31.314921 30051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:31.314957 30051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:31.319988 30051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:31.321009 30051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:31.322005 30051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:31.330240 30065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:31.330268 30060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:31.330245 30059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:31.330510 30051 server_base.cc:1034] running on GCE node
I20260812 11:15:31.331091 30051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:31.332201 30051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:31.333384 30051 hybrid_clock.cc:648] HybridClock initialized: now 1786533331333363 us; error 33 us; skew 500 ppm
Aug 12 11:15:31 dist-test-slave-zpvx krb5kdc[30023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:31.336226 30051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:31.337172 30051 webserver.cc:460] Webserver started at http://127.0.0.1:45823/ using document root <none> and password file <none>
I20260812 11:15:31.337723 30051 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:31.337785 30051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:31.338006 30051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:31.339982 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5"
format_stamp: "Formatted at 2026-08-12 11:15:31 on dist-test-slave-zpvx"
server_key: "624fb085c7682a3355054766192997b3"
server_key_iv: "2b1995ceb1376f82a1d2f49b788e43ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:31.340587 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5"
format_stamp: "Formatted at 2026-08-12 11:15:31 on dist-test-slave-zpvx"
server_key: "624fb085c7682a3355054766192997b3"
server_key_iv: "2b1995ceb1376f82a1d2f49b788e43ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:31.344177 30051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260812 11:15:31.346603 30080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:31.347780 30051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:31.347925 30051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5"
format_stamp: "Formatted at 2026-08-12 11:15:31 on dist-test-slave-zpvx"
server_key: "624fb085c7682a3355054766192997b3"
server_key_iv: "2b1995ceb1376f82a1d2f49b788e43ae"
server_key_version: "encryptionkey@0"
I20260812 11:15:31.348039 30051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:31.361799 30051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:31.364893 30051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:31.365087 30051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:31.372125 30051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42265
I20260812 11:15:31.372143 30171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42265 every 8 connection(s)
I20260812 11:15:31.373095 30051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:31.375303 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30051
I20260812 11:15:31.375471 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:31.375816 23943 external_mini_cluster.cc:1428] Setting key 48659aafed4200197f2f6d4c3303bd99
I20260812 11:15:31.376250 30172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:31.382839 30172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5: Bootstrap starting.
I20260812 11:15:31.384702 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:31.377425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55629 (local address 127.0.0.1:42265)
0812 11:15:31.377896 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:31.377910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:31.377945 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:31.378595 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:31.378612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:31.378929 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:31.379271 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:31.379649 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:31.380681 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:31.381581 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:31.382041 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:31.382228 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:31.383070 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:31.383075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:31.383086 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:31.383453 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:31.383505 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:31.383528 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:31.383637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:31.383761 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 11:15:31.383962 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":163,"threads_started":1}
I20260812 11:15:31.385435 30172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:31.386252 30172 log.cc:824] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:31.387655 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:31.388445 30172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5: No bootstrap required, opened a new log
I20260812 11:15:31.393270 30172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260812 11:15:31.393570 30172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:31.393677 30172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4b28e7bedb494b836e80ef2cf2dfa5, State: Initialized, Role: FOLLOWER
I20260812 11:15:31.393783 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:31.389661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40199 (local address 127.0.0.1:42265)
0812 11:15:31.389780 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:31.389784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:31.389949 (+ 165us) server_negotiation.cc:407] Connection header received
0812 11:15:31.390081 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:31.390083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:31.390117 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:31.390181 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:31.390489 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:31.390934 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:31.392140 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:31.392346 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:31.392479 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:31.393247 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:31.393250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:31.393252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:31.393263 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:31.393286 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:31.393295 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:31.393353 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:31.393487 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 11:15:31.393536 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:15:31.394240 30172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260812 11:15:31.394410 30172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:31.394483 30172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:31.394594 30172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:31.395591 30172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260812 11:15:31.395900 30172 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a4b28e7bedb494b836e80ef2cf2dfa5; no voters:
I20260812 11:15:31.396152 30172 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:31.396253 30194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:31.396512 30194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4b28e7bedb494b836e80ef2cf2dfa5, State: Running, Role: LEADER
I20260812 11:15:31.396864 30194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260812 11:15:31.397476 30172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:31.398495 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } } }
I20260812 11:15:31.398624 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:31.399019 30209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:31.398978 30197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4b28e7bedb494b836e80ef2cf2dfa5. Latest consensus state: current_term: 1 leader_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4b28e7bedb494b836e80ef2cf2dfa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } } }
I20260812 11:15:31.399953 30197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:31.402124 30209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:31.407567 30209 catalog_manager.cc:1347] Generated new cluster ID: e516ea255a83495e8f5a67eb08ef7943
I20260812 11:15:31.407629 30209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:31.418944 30209 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:31.420073 30209 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:31.432919 30209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4b28e7bedb494b836e80ef2cf2dfa5: Generated new TSK 0
I20260812 11:15:31.433616 30209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:31.438472 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30051
2026-08-12T11:15:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1691 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:31 dist-test-slave-zpvx krb5kdc[30230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:31 dist-test-slave-zpvx krb5kdc[30230](info): set up 2 sockets
Aug 12 11:15:31 dist-test-slave-zpvx krb5kdc[30230](info): commencing operation
krb5kdc: starting...
W20260812 11:15:32.788601 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:32 dist-test-slave-zpvx krb5kdc[30230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:32.914989 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:33.030647 30252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:33.030982 30252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:33.031052 30252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:33.035111 30252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:33.035167 30252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:33.035194 30252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:33.035211 30252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:33.035262 30252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:33.040362 30252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:33.041499 30252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:33.042527 30252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:33.051034 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.051057 30259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:33.050984 30260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:33.051349 30252 server_base.cc:1034] running on GCE node
I20260812 11:15:33.051893 30252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:33.053020 30252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:33.054205 30252 hybrid_clock.cc:648] HybridClock initialized: now 1786533333054150 us; error 44 us; skew 500 ppm
Aug 12 11:15:33 dist-test-slave-zpvx krb5kdc[30230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:33.057044 30252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:33.058020 30252 webserver.cc:460] Webserver started at http://127.0.0.1:34281/ using document root <none> and password file <none>
I20260812 11:15:33.058653 30252 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:33.058756 30252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:33.059000 30252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:33.061295 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "5b4299381d54431e9337af6c724c33c2"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-zpvx"
server_key: "fab37c080b068e4e70902b70ceb943aa"
server_key_iv: "a65d2374ec960601c71d0d4e41d925b5"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.061916 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "5b4299381d54431e9337af6c724c33c2"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-zpvx"
server_key: "fab37c080b068e4e70902b70ceb943aa"
server_key_iv: "a65d2374ec960601c71d0d4e41d925b5"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.065850 30252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:15:33.068434 30275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:33.069653 30252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:33.069804 30252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "5b4299381d54431e9337af6c724c33c2"
format_stamp: "Formatted at 2026-08-12 11:15:33 on dist-test-slave-zpvx"
server_key: "fab37c080b068e4e70902b70ceb943aa"
server_key_iv: "a65d2374ec960601c71d0d4e41d925b5"
server_key_version: "encryptionkey@0"
I20260812 11:15:33.069912 30252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:33.074558 30252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:33.077798 30252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:33.077986 30252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:33.085095 30252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45457
I20260812 11:15:33.085124 30361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45457 every 8 connection(s)
I20260812 11:15:33.086040 30252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:33.088930 30364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:33.091892 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30252
I20260812 11:15:33.092047 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:33.092443 23943 external_mini_cluster.cc:1428] Setting key d0995622212ca4645aba015ae4936980
I20260812 11:15:33.094928 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: Bootstrap starting.
I20260812 11:15:33.097684 30364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:33.098358 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.094007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39729 (local address 127.0.0.1:45457)
0812 11:15:33.094476 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.094515 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.094555 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:15:33.095317 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.095337 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.095651 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.095961 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.096920 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.096925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.096939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:33.096951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.097013 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:33.097034 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.097138 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.097274 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.097470 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":166,"threads_started":1}
I20260812 11:15:33.098503 30364 log.cc:824] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:33.100801 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: No bootstrap required, opened a new log
I20260812 11:15:33.101965 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:33.104584 30364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260812 11:15:33.104907 30364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:33.104980 30364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4299381d54431e9337af6c724c33c2, State: Initialized, Role: FOLLOWER
I20260812 11:15:33.105466 30364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260812 11:15:33.105652 30364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:33.105741 30364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:33.105868 30364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:33.105912 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:33.104137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33803 (local address 127.0.0.1:45457)
0812 11:15:33.104282 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:33.104287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:33.104516 (+ 229us) server_negotiation.cc:407] Connection header received
0812 11:15:33.104647 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:33.104649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:33.104681 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:33.104753 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:33.105494 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:33.105497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:33.105499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:33.105510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:33.105536 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:33.105547 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:33.105645 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:33.105732 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 11:15:33.105773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:15:33.106940 30364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260812 11:15:33.107237 30364 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4299381d54431e9337af6c724c33c2; no voters:
I20260812 11:15:33.107486 30364 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:33.107620 30390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:33.107905 30390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4299381d54431e9337af6c724c33c2, State: Running, Role: LEADER
I20260812 11:15:33.108244 30390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260812 11:15:33.108755 30364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:33.110081 30391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4299381d54431e9337af6c724c33c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260812 11:15:33.110231 30391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:33.110610 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4299381d54431e9337af6c724c33c2. Latest consensus state: current_term: 1 leader_uuid: "5b4299381d54431e9337af6c724c33c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4299381d54431e9337af6c724c33c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260812 11:15:33.110718 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2 [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:33.113722 30421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:33.113807 30421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:33.113885 30411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:33.114620 30411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:33.120191 30411 catalog_manager.cc:1347] Generated new cluster ID: 16907abc7b1944bfa3e84e3e60ec14ea
I20260812 11:15:33.120265 30411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:33.133066 30411 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:33.133438 30411 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:33.143764 30411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4299381d54431e9337af6c724c33c2: Generated new TSK 0
I20260812 11:15:33.144605 30411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:33.148569 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30252
2026-08-12T11:15:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1702 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:33 dist-test-slave-zpvx krb5kdc[30427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:33 dist-test-slave-zpvx krb5kdc[30427](info): set up 2 sockets
Aug 12 11:15:33 dist-test-slave-zpvx krb5kdc[30427](info): commencing operation
krb5kdc: starting...
W20260812 11:15:34.446851 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:34.570883 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:34.682116 30446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:34.682375 30446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:34.682417 30446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:34.687567 30446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:34.687659 30446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:34.687691 30446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:34.687717 30446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:34.687744 30446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:34.695359 30446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:34.696727 30446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:34.698215 30446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:34.706662 30455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:34.706646 30453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:34.706694 30461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:34.707001 30446 server_base.cc:1034] running on GCE node
I20260812 11:15:34.707677 30446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:34.709151 30446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:34.710350 30446 hybrid_clock.cc:648] HybridClock initialized: now 1786533334710271 us; error 94 us; skew 500 ppm
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:34.717523 30446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:34.718451 30446 webserver.cc:460] Webserver started at http://127.0.0.1:35903/ using document root <none> and password file <none>
I20260812 11:15:34.719040 30446 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:34.719129 30446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:34.719365 30446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:34.721482 30446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "63098959a4934aa082526b5aa6adc360"
format_stamp: "Formatted at 2026-08-12 11:15:34 on dist-test-slave-zpvx"
server_key: "308ba141cb30e848bd72b0715e719cf6"
server_key_iv: "556affe59b79a4cd48f2ba2cd0bb5f98"
server_key_version: "encryptionkey@0"
I20260812 11:15:34.722062 30446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "63098959a4934aa082526b5aa6adc360"
format_stamp: "Formatted at 2026-08-12 11:15:34 on dist-test-slave-zpvx"
server_key: "308ba141cb30e848bd72b0715e719cf6"
server_key_iv: "556affe59b79a4cd48f2ba2cd0bb5f98"
server_key_version: "encryptionkey@0"
I20260812 11:15:34.725802 30446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:15:34.728343 30472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:34.729600 30446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:34.729741 30446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "63098959a4934aa082526b5aa6adc360"
format_stamp: "Formatted at 2026-08-12 11:15:34 on dist-test-slave-zpvx"
server_key: "308ba141cb30e848bd72b0715e719cf6"
server_key_iv: "556affe59b79a4cd48f2ba2cd0bb5f98"
server_key_version: "encryptionkey@0"
I20260812 11:15:34.729848 30446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:34.752067 30446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:34.755214 30446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:34.755440 30446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:34.763309 30446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41105
I20260812 11:15:34.763337 30560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41105 every 8 connection(s)
I20260812 11:15:34.764269 30446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:34.767197 30561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:34.767771 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30446
I20260812 11:15:34.767925 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:34.768224 23943 external_mini_cluster.cc:1428] Setting key 1aa18b6be11ac26297589a5b745bb6dc
I20260812 11:15:34.775033 30561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360: Bootstrap starting.
I20260812 11:15:34.777645 30561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360: Neither blocks nor log segments found. Creating new log.
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:15:34.778704 30561 log.cc:824] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:34.780977 30561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360: No bootstrap required, opened a new log
I20260812 11:15:34.783614 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.769698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:48471 (local address 127.0.0.1:41105)
0812 11:15:34.770167 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.770187 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.770234 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:15:34.771006 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.771030 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.771528 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.771914 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:34.772677 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.774368 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.775221 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.775680 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.775862 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.778906 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:34.778912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:34.778925 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:34.778951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:34.780714 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.781322 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.781327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.781331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.781438 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.781812 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.781814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.781815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.782169 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:34.782323 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.782666 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.782897 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260812 11:15:34.785511 30561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260812 11:15:34.785825 30561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:34.785912 30561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63098959a4934aa082526b5aa6adc360, State: Initialized, Role: FOLLOWER
I20260812 11:15:34.786424 30561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260812 11:15:34.786595 30561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:34.786698 30561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:34.786820 30561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:34.786875 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:34.787811 30561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260812 11:15:34.788190 30561 leader_election.cc:302] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63098959a4934aa082526b5aa6adc360; no voters:
I20260812 11:15:34.788564 30561 leader_election.cc:288] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:34.788640 30579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:34.788880 30579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [term 1 LEADER]: Becoming Leader. State: Replica: 63098959a4934aa082526b5aa6adc360, State: Running, Role: LEADER
I20260812 11:15:34.789274 30579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } }
I20260812 11:15:34.789996 30561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:34.791036 30580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63098959a4934aa082526b5aa6adc360. Latest consensus state: current_term: 1 leader_uuid: "63098959a4934aa082526b5aa6adc360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } } }
I20260812 11:15:34.791198 30580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:34.791452 30580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63098959a4934aa082526b5aa6adc360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63098959a4934aa082526b5aa6adc360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41105 } } }
I20260812 11:15:34.791545 30580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:34.792786 30593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:34.796307 30593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:34.797768 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:34.789184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:53359 (local address 127.0.0.1:41105)
0812 11:15:34.789296 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:34.789300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:34.789558 (+ 258us) server_negotiation.cc:407] Connection header received
0812 11:15:34.789684 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:34.789687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:34.789732 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:34.789821 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:34.790614 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.791372 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.792488 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:34.792829 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:34.792914 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:34.794452 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:34.794456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:34.794459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:34.794483 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:34.795946 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.796504 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.796507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.796508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.796580 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:34.796912 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:34.796916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:34.796918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:34.797088 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:34.797237 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:34.797502 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 11:15:34.797595 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 11:15:34.802812 30593 catalog_manager.cc:1347] Generated new cluster ID: d1d711df1b8942fbb9e68e72fa0e6597
I20260812 11:15:34.802883 30593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:34.820881 30593 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:34.821789 30593 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:34.825389 30593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63098959a4934aa082526b5aa6adc360: Generated new TSK 0
I20260812 11:15:34.826062 30593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:34.830008 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30446
W20260812 11:15:34.848918 30425 connection.cc:535] client connection to 127.0.0.1:41105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41105): Connection reset by peer (error 104)
2026-08-12T11:15:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1690 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30611](info): set up 2 sockets
Aug 12 11:15:34 dist-test-slave-zpvx krb5kdc[30611](info): commencing operation
krb5kdc: starting...
W20260812 11:15:36.198338 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.335s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:36 dist-test-slave-zpvx krb5kdc[30611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:36.328022 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:36.440325 30646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:36.440630 30646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:36.440701 30646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:36.444921 30646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:36.445025 30646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:36.445063 30646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:36.445116 30646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:36.445168 30646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:36.450513 30646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:36.451573 30646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:36.452677 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:36.461688 30655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:36.461731 30652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:36.461797 30646 server_base.cc:1034] running on GCE node
W20260812 11:15:36.461697 30657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:36.462611 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:36.463728 30646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:36.464929 30646 hybrid_clock.cc:648] HybridClock initialized: now 1786533336464900 us; error 40 us; skew 500 ppm
Aug 12 11:15:36 dist-test-slave-zpvx krb5kdc[30611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:36.467666 30646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:36.468621 30646 webserver.cc:460] Webserver started at http://127.0.0.1:39151/ using document root <none> and password file <none>
I20260812 11:15:36.469157 30646 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:36.469223 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:36.469408 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:36.471431 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "ef78e5b6d69244a8a3172e9174a60ccd"
format_stamp: "Formatted at 2026-08-12 11:15:36 on dist-test-slave-zpvx"
server_key: "0170e5d8e2d647280b2ce9934f8aefba"
server_key_iv: "595937417c007d538ef245f6e58838bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:36.471956 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "ef78e5b6d69244a8a3172e9174a60ccd"
format_stamp: "Formatted at 2026-08-12 11:15:36 on dist-test-slave-zpvx"
server_key: "0170e5d8e2d647280b2ce9934f8aefba"
server_key_iv: "595937417c007d538ef245f6e58838bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:36.475677 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:15:36.478137 30670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:36.479359 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:36.479458 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "ef78e5b6d69244a8a3172e9174a60ccd"
format_stamp: "Formatted at 2026-08-12 11:15:36 on dist-test-slave-zpvx"
server_key: "0170e5d8e2d647280b2ce9934f8aefba"
server_key_iv: "595937417c007d538ef245f6e58838bc"
server_key_version: "encryptionkey@0"
I20260812 11:15:36.479539 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:36.514452 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:36.517328 30646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:36.517498 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:36.524989 30646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33415
I20260812 11:15:36.525015 30744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33415 every 8 connection(s)
I20260812 11:15:36.525935 30646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:36.528699 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:36.534266 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd: Bootstrap starting.
I20260812 11:15:36.534595 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30646
I20260812 11:15:36.534736 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:36.534998 23943 external_mini_cluster.cc:1428] Setting key 2b5acff2c8fc6d022106c3b965a0c590
I20260812 11:15:36.536932 30747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:36.537801 30747 log.cc:824] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:36.539934 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd: No bootstrap required, opened a new log
I20260812 11:15:36.544447 30747 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260812 11:15:36.544699 30747 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:36.544757 30747 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef78e5b6d69244a8a3172e9174a60ccd, State: Initialized, Role: FOLLOWER
I20260812 11:15:36.545015 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:36.536324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:50061 (local address 127.0.0.1:33415)
0812 11:15:36.536782 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:36.536803 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:36.536848 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:15:36.537724 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:36.537747 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:36.538177 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:36.538457 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:36.538892 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:36.540309 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:36.541203 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:36.541911 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:36.542060 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:36.542976 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:36.542984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:36.543005 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:36.543500 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:36.543564 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:36.543595 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:36.543742 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:36.543911 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 11:15:36.544038 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
I20260812 11:15:36.545239 30747 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260812 11:15:36.545408 30747 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:36.545476 30747 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:36.545575 30747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:36.546555 30747 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260812 11:15:36.546923 30747 leader_election.cc:302] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef78e5b6d69244a8a3172e9174a60ccd; no voters:
I20260812 11:15:36.547274 30747 leader_election.cc:288] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:36.547369 30754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:36.547622 30754 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [term 1 LEADER]: Becoming Leader. State: Replica: ef78e5b6d69244a8a3172e9174a60ccd, State: Running, Role: LEADER
I20260812 11:15:36.547930 30754 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260812 11:15:36.548056 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:36.548544 30747 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:36.549140 30755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef78e5b6d69244a8a3172e9174a60ccd. Latest consensus state: current_term: 1 leader_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } } }
I20260812 11:15:36.549275 30755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:36.549616 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef78e5b6d69244a8a3172e9174a60ccd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } } }
I20260812 11:15:36.549738 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:36.550278 30765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:36.553408 30765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:36.555701 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:36.551456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:49785 (local address 127.0.0.1:33415)
0812 11:15:36.551557 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:36.551561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:36.551574 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:36.551607 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:36.551610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:36.551648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:36.551733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:36.552007 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:36.552771 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:36.553858 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:36.554189 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:36.554308 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:36.555171 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:36.555175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:36.555177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:36.555190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:36.555218 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:36.555231 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:36.555313 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:36.555519 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 11:15:36.555555 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 11:15:36.559074 30765 catalog_manager.cc:1347] Generated new cluster ID: 2a688162dda0474ba41c2691159c6546
I20260812 11:15:36.559145 30765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:36.574208 30765 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:36.575119 30765 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:36.580672 30765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef78e5b6d69244a8a3172e9174a60ccd: Generated new TSK 0
I20260812 11:15:36.581408 30765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:36.592808 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30646
W20260812 11:15:36.606885 30606 connection.cc:535] client connection to 127.0.0.1:33415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33415): Connection reset by peer (error 104)
2026-08-12T11:15:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1757 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:36 dist-test-slave-zpvx krb5kdc[30803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:36 dist-test-slave-zpvx krb5kdc[30803](info): set up 2 sockets
Aug 12 11:15:36 dist-test-slave-zpvx krb5kdc[30803](info): commencing operation
krb5kdc: starting...
W20260812 11:15:37.912753 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:38 dist-test-slave-zpvx krb5kdc[30803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:38.048252 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:38.163060 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:38.163389 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:38.163475 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:38.167635 30823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:38.167740 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:38.167781 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:38.167835 30823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:38.167873 30823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:38.173118 30823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:38.174201 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:38.175275 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:38.184106 30833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:38.184113 30832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:38.184194 30823 server_base.cc:1034] running on GCE node
W20260812 11:15:38.184160 30839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:38.184916 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:38.186008 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:38.187214 30823 hybrid_clock.cc:648] HybridClock initialized: now 1786533338187199 us; error 34 us; skew 500 ppm
W20260812 11:15:38.392397 30823 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-zpvx': real 0.205s user 0.001s sys 0.000s
Aug 12 11:15:38 dist-test-slave-zpvx krb5kdc[30803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:38.395633 30823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:38.397264 30823 webserver.cc:460] Webserver started at http://127.0.0.1:33953/ using document root <none> and password file <none>
I20260812 11:15:38.398260 30823 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:38.398501 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:38.399008 30823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:38.402501 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d"
format_stamp: "Formatted at 2026-08-12 11:15:38 on dist-test-slave-zpvx"
server_key: "41b16270e9af2a9b0d38fabc2462e3cc"
server_key_iv: "f66975569420e9d3d874c963ef372fab"
server_key_version: "encryptionkey@0"
I20260812 11:15:38.403378 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d"
format_stamp: "Formatted at 2026-08-12 11:15:38 on dist-test-slave-zpvx"
server_key: "41b16270e9af2a9b0d38fabc2462e3cc"
server_key_iv: "f66975569420e9d3d874c963ef372fab"
server_key_version: "encryptionkey@0"
I20260812 11:15:38.414207 30823 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.003s
I20260812 11:15:38.417797 30852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:38.419559 30823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:38.419762 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d"
format_stamp: "Formatted at 2026-08-12 11:15:38 on dist-test-slave-zpvx"
server_key: "41b16270e9af2a9b0d38fabc2462e3cc"
server_key_iv: "f66975569420e9d3d874c963ef372fab"
server_key_version: "encryptionkey@0"
I20260812 11:15:38.419991 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:38.427801 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:38.432745 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:38.432940 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:38.453783 30823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32881
I20260812 11:15:38.453815 30940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32881 every 8 connection(s)
I20260812 11:15:38.455677 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:38.459939 30944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:38.466866 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 30823
I20260812 11:15:38.467058 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:38.467542 23943 external_mini_cluster.cc:1428] Setting key 6b9b485ac38500b12712d0960e48c9e6
I20260812 11:15:38.468825 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d: Bootstrap starting.
I20260812 11:15:38.471958 30944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:38.473011 30944 log.cc:824] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:38.475433 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:38.469542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:59849 (local address 127.0.0.1:32881)
0812 11:15:38.470042 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:38.470065 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:38.470115 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:15:38.471101 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:38.471129 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:38.471558 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:38.472047 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:38.474039 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:38.474045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:38.474062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:38.474076 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:38.474148 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:38.474209 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:38.474369 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:38.474496 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:15:38.474680 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":138,"threads_started":1}
I20260812 11:15:38.475643 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d: No bootstrap required, opened a new log
I20260812 11:15:38.478571 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:38.481242 30944 raft_consensus.cc:348] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260812 11:15:38.481556 30944 raft_consensus.cc:374] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:38.481617 30944 raft_consensus.cc:729] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a241e74c32ca4ecaa40ecc9dc32d7d7d, State: Initialized, Role: FOLLOWER
I20260812 11:15:38.482163 30944 consensus_queue.cc:260] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260812 11:15:38.482355 30944 raft_consensus.cc:388] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:38.482437 30944 raft_consensus.cc:482] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:38.482560 30944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:38.483450 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:38.481463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:50075 (local address 127.0.0.1:32881)
0812 11:15:38.481590 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:38.481595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:38.481880 (+ 285us) server_negotiation.cc:407] Connection header received
0812 11:15:38.482097 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:38.482101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:38.482160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:38.482239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:38.482989 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:38.482993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:38.482995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:38.483009 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:38.483039 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:38.483058 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:38.483167 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:38.483268 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 11:15:38.483316 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 11:15:38.484056 30944 raft_consensus.cc:504] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260812 11:15:38.484532 30944 leader_election.cc:302] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a241e74c32ca4ecaa40ecc9dc32d7d7d; no voters:
I20260812 11:15:38.484923 30944 leader_election.cc:288] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:38.485033 30962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:38.485265 30962 raft_consensus.cc:686] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [term 1 LEADER]: Becoming Leader. State: Replica: a241e74c32ca4ecaa40ecc9dc32d7d7d, State: Running, Role: LEADER
I20260812 11:15:38.485631 30962 consensus_queue.cc:237] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260812 11:15:38.486567 30944 sys_catalog.cc:564] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:38.487634 30965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a241e74c32ca4ecaa40ecc9dc32d7d7d. Latest consensus state: current_term: 1 leader_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } } }
I20260812 11:15:38.487918 30963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241e74c32ca4ecaa40ecc9dc32d7d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } } }
I20260812 11:15:38.488063 30963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:38.488107 30965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:38.489104 30978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:38.491853 30978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:38.497880 30978 catalog_manager.cc:1347] Generated new cluster ID: 96fe26d2cf604b37baf10f090f717088
I20260812 11:15:38.497946 30978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:38.511693 30978 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:38.512061 30978 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:38.519129 30978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a241e74c32ca4ecaa40ecc9dc32d7d7d: Generated new TSK 0
I20260812 11:15:38.519811 30978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:38.529371 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 30823
2026-08-12T11:15:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:38 dist-test-slave-zpvx krb5kdc[30998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:38 dist-test-slave-zpvx krb5kdc[30998](info): set up 2 sockets
Aug 12 11:15:38 dist-test-slave-zpvx krb5kdc[30998](info): commencing operation
krb5kdc: starting...
W20260812 11:15:39.860108 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:39 dist-test-slave-zpvx krb5kdc[30998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:39.991935 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:40.105742 31021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:40.106048 31021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:40.106124 31021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:40.110106 31021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:40.110220 31021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:40.110262 31021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:40.110301 31021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:40.110332 31021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:40.115502 31021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:40.116686 31021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:40.117849 31021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:40.126884 31040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:40.126879 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:40.127225 31021 server_base.cc:1034] running on GCE node
W20260812 11:15:40.126889 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:40.127780 31021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:40.129004 31021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:40.130261 31021 hybrid_clock.cc:648] HybridClock initialized: now 1786533340130222 us; error 53 us; skew 500 ppm
Aug 12 11:15:40 dist-test-slave-zpvx krb5kdc[30998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:40.138720 31021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:40.139780 31021 webserver.cc:460] Webserver started at http://127.0.0.1:39295/ using document root <none> and password file <none>
I20260812 11:15:40.140340 31021 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:40.140484 31021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:40.140686 31021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:40.142817 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "0c89004938c8473a88c016129c1d493f"
format_stamp: "Formatted at 2026-08-12 11:15:40 on dist-test-slave-zpvx"
server_key: "1011776012769688536a6375da7d5a7b"
server_key_iv: "b7c9172f9eee82086a310ba368d8376e"
server_key_version: "encryptionkey@0"
I20260812 11:15:40.143359 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "0c89004938c8473a88c016129c1d493f"
format_stamp: "Formatted at 2026-08-12 11:15:40 on dist-test-slave-zpvx"
server_key: "1011776012769688536a6375da7d5a7b"
server_key_iv: "b7c9172f9eee82086a310ba368d8376e"
server_key_version: "encryptionkey@0"
I20260812 11:15:40.147171 31021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:40.149803 31057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:40.151088 31021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:40.151223 31021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "0c89004938c8473a88c016129c1d493f"
format_stamp: "Formatted at 2026-08-12 11:15:40 on dist-test-slave-zpvx"
server_key: "1011776012769688536a6375da7d5a7b"
server_key_iv: "b7c9172f9eee82086a310ba368d8376e"
server_key_version: "encryptionkey@0"
I20260812 11:15:40.151314 31021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:40.202782 31021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:40.206035 31021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:40.206214 31021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:40.213814 31021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38755
I20260812 11:15:40.213843 31157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38755 every 8 connection(s)
I20260812 11:15:40.214783 31021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:40.217676 31158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:40.219164 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31021
I20260812 11:15:40.219312 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:40.219599 23943 external_mini_cluster.cc:1428] Setting key 3a3b5d4a385cbca27940495ff0577051
I20260812 11:15:40.223719 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: Bootstrap starting.
I20260812 11:15:40.225993 31158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:40.226724 31158 log.cc:824] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:15:40 dist-test-slave-zpvx krb5kdc[30998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:15:40.228768 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: No bootstrap required, opened a new log
I20260812 11:15:40.232482 31158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260812 11:15:40.232745 31158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:40.232813 31158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c89004938c8473a88c016129c1d493f, State: Initialized, Role: FOLLOWER
I20260812 11:15:40.233320 31158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260812 11:15:40.233480 31158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:40.233544 31158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:40.233639 31158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:40.234341 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:40.221104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:58801 (local address 127.0.0.1:38755)
0812 11:15:40.221536 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:40.221552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:40.221587 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:40.222225 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:40.222241 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:40.222562 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:40.222912 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:40.223776 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:40.224865 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:40.225764 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:40.226287 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:40.226463 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:40.229223 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:40.229228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:40.229241 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:40.229267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:40.230942 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:40.231512 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:40.231523 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:40.231527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:40.231641 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:40.232021 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:40.232025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:40.232027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:40.232523 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:40.232766 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:40.233090 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 11:15:40.233332 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260812 11:15:40.234792 31158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260812 11:15:40.235148 31158 leader_election.cc:302] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c89004938c8473a88c016129c1d493f; no voters:
I20260812 11:15:40.235510 31158 leader_election.cc:288] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:40.235661 31170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:40.235883 31170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [term 1 LEADER]: Becoming Leader. State: Replica: 0c89004938c8473a88c016129c1d493f, State: Running, Role: LEADER
I20260812 11:15:40.236174 31170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260812 11:15:40.236881 31158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:40.237409 31170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c89004938c8473a88c016129c1d493f. Latest consensus state: current_term: 1 leader_uuid: "0c89004938c8473a88c016129c1d493f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } } }
I20260812 11:15:40.237530 31170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:40.237975 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:40.239697 31175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c89004938c8473a88c016129c1d493f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c89004938c8473a88c016129c1d493f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } } }
I20260812 11:15:40.239799 31175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f [sys.catalog]: This master's current role is: LEADER
W20260812 11:15:40.242472 31201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:40.242555 31201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:40.242631 31179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:40.243343 31179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:40.249476 31179 catalog_manager.cc:1347] Generated new cluster ID: f76aa2b6fb4e47d1985847b0c2e99d84
I20260812 11:15:40.249543 31179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:40.251487 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:40.240593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:43403 (local address 127.0.0.1:38755)
0812 11:15:40.240724 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:40.240728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:40.240918 (+ 190us) server_negotiation.cc:407] Connection header received
0812 11:15:40.241062 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:40.241065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:40.241113 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:40.241218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:40.241878 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:40.242393 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:40.245650 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:40.245896 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:40.246018 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:40.247602 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:40.247606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:40.247608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:40.247639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:40.249543 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:40.250079 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:40.250083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:40.250085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:40.250178 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:40.250551 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:40.250554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:40.250556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:40.250744 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:40.250933 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:40.251222 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:15:40.251318 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:40.269629 31179 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:40.270876 31179 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:40.283578 31179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c89004938c8473a88c016129c1d493f: Generated new TSK 0
I20260812 11:15:40.284353 31179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:40.288556 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31021
W20260812 11:15:40.301745 30995 connection.cc:535] client connection to 127.0.0.1:38755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38755): Connection reset by peer (error 104)
2026-08-12T11:15:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:40 dist-test-slave-zpvx krb5kdc[31214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:40 dist-test-slave-zpvx krb5kdc[31214](info): set up 2 sockets
Aug 12 11:15:40 dist-test-slave-zpvx krb5kdc[31214](info): commencing operation
krb5kdc: starting...
W20260812 11:15:41.597232 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:41 dist-test-slave-zpvx krb5kdc[31214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:41.723779 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:41.835995 31239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:41.836292 31239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:41.836417 31239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:41.840349 31239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:41.840461 31239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:41.840500 31239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:41.840533 31239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:41.840564 31239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:41.845692 31239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:41.846704 31239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:41.847738 31239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:41.856164 31248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:41.856280 31250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:41.856184 31247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:41.856405 31239 server_base.cc:1034] running on GCE node
I20260812 11:15:41.857004 31239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:41.858096 31239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:41.859269 31239 hybrid_clock.cc:648] HybridClock initialized: now 1786533341859240 us; error 40 us; skew 500 ppm
Aug 12 11:15:41 dist-test-slave-zpvx krb5kdc[31214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:41.862133 31239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:41.862998 31239 webserver.cc:460] Webserver started at http://127.0.0.1:33805/ using document root <none> and password file <none>
I20260812 11:15:41.863567 31239 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:41.863636 31239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:41.863885 31239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:41.866024 31239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "c5ee76bc5ffa4c5184f458751197ae69"
format_stamp: "Formatted at 2026-08-12 11:15:41 on dist-test-slave-zpvx"
server_key: "d553e69cf7aa2eb1def30c13a9384a5d"
server_key_iv: "e3a098b466cc8a6594705cd7e567d5cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:41.866611 31239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "c5ee76bc5ffa4c5184f458751197ae69"
format_stamp: "Formatted at 2026-08-12 11:15:41 on dist-test-slave-zpvx"
server_key: "d553e69cf7aa2eb1def30c13a9384a5d"
server_key_iv: "e3a098b466cc8a6594705cd7e567d5cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:41.870466 31239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:15:41.873008 31263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:41.874327 31239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:41.874490 31239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c5ee76bc5ffa4c5184f458751197ae69"
format_stamp: "Formatted at 2026-08-12 11:15:41 on dist-test-slave-zpvx"
server_key: "d553e69cf7aa2eb1def30c13a9384a5d"
server_key_iv: "e3a098b466cc8a6594705cd7e567d5cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:41.874614 31239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:41.900416 31239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:41.903874 31239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:41.904095 31239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:41.911463 31239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34133
I20260812 11:15:41.911479 31355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34133 every 8 connection(s)
I20260812 11:15:41.912494 31239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:41.915269 31356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:41.919932 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31239
I20260812 11:15:41.920046 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:41.920359 23943 external_mini_cluster.cc:1428] Setting key ff79ccb6dd80049bf4d9263983126077
I20260812 11:15:41.920964 31356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69: Bootstrap starting.
I20260812 11:15:41.923554 31356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:41.924456 31356 log.cc:824] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:41.926523 31356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69: No bootstrap required, opened a new log
I20260812 11:15:41.930867 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:41.921907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:41931 (local address 127.0.0.1:34133)
0812 11:15:41.922334 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:41.922352 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:41.922394 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:15:41.923195 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:41.923218 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:41.923668 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:41.924033 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:41.924464 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:41.925806 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:41.926905 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:41.927563 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:41.927728 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:41.928727 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:41.928733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:41.928749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:41.929302 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:41.929385 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:41.929421 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:41.929582 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:41.929806 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 11:15:41.929938 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260812 11:15:41.930962 31356 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260812 11:15:41.931231 31356 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:41.931298 31356 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ee76bc5ffa4c5184f458751197ae69, State: Initialized, Role: FOLLOWER
I20260812 11:15:41.931692 31356 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260812 11:15:41.931854 31356 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:41.931947 31356 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:41.932049 31356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:41.933171 31356 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260812 11:15:41.933524 31356 leader_election.cc:302] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ee76bc5ffa4c5184f458751197ae69; no voters:
I20260812 11:15:41.933890 31356 leader_election.cc:288] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:41.934032 31361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:41.934285 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:41.934315 31361 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [term 1 LEADER]: Becoming Leader. State: Replica: c5ee76bc5ffa4c5184f458751197ae69, State: Running, Role: LEADER
I20260812 11:15:41.934623 31361 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260812 11:15:41.935160 31356 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:41.936169 31363 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ee76bc5ffa4c5184f458751197ae69. Latest consensus state: current_term: 1 leader_uuid: "c5ee76bc5ffa4c5184f458751197ae69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } } }
I20260812 11:15:41.936296 31363 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:41.936671 31362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ee76bc5ffa4c5184f458751197ae69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ee76bc5ffa4c5184f458751197ae69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } } }
I20260812 11:15:41.936733 31390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:41.936765 31362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:41.940219 31390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:41.942574 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:41.937124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:44707 (local address 127.0.0.1:34133)
0812 11:15:41.937278 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:41.937282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:41.938122 (+ 840us) server_negotiation.cc:407] Connection header received
0812 11:15:41.938222 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:41.938224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:41.938271 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:41.938352 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:41.938714 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:41.939486 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:41.940692 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:41.941084 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:41.941217 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:41.942098 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:41.942102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:41.942104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:41.942117 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:41.942148 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:41.942159 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:41.942264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:41.942391 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:15:41.942423 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:15:41.945686 31390 catalog_manager.cc:1347] Generated new cluster ID: 28c3af40fc6e482fbc3eb84538b049a2
I20260812 11:15:41.945780 31390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:41.955886 31390 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:41.956772 31390 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:41.963002 31390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ee76bc5ffa4c5184f458751197ae69: Generated new TSK 0
I20260812 11:15:41.963610 31390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:41.969267 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31239
W20260812 11:15:41.992733 31211 connection.cc:535] client connection to 127.0.0.1:34133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34133): Connection reset by peer (error 104)
2026-08-12T11:15:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1692 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:42 dist-test-slave-zpvx krb5kdc[31414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:42 dist-test-slave-zpvx krb5kdc[31414](info): set up 2 sockets
Aug 12 11:15:42 dist-test-slave-zpvx krb5kdc[31414](info): commencing operation
krb5kdc: starting...
W20260812 11:15:43.295042 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:43 dist-test-slave-zpvx krb5kdc[31414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:15:43.424260 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:43.536171 31436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:43.536526 31436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:43.536615 31436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:43.540504 31436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:43.540597 31436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:43.540638 31436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:43.540670 31436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:43.540696 31436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:43.545825 31436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:43.546851 31436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:43.547860 31436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:43.556494 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:43.556526 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:43.556694 31436 server_base.cc:1034] running on GCE node
W20260812 11:15:43.556494 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:43.557351 31436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:43.558396 31436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:43.559561 31436 hybrid_clock.cc:648] HybridClock initialized: now 1786533343559533 us; error 41 us; skew 500 ppm
Aug 12 11:15:43 dist-test-slave-zpvx krb5kdc[31414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:43.562695 31436 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:15:43.563578 31436 webserver.cc:460] Webserver started at http://127.0.0.1:42727/ using document root <none> and password file <none>
I20260812 11:15:43.564177 31436 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:43.564241 31436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:43.564533 31436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:43.566546 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "20a72ba99edd41d0a8c694806493a6b0"
format_stamp: "Formatted at 2026-08-12 11:15:43 on dist-test-slave-zpvx"
server_key: "2dc1ccb358116abadca3e177b9dfb712"
server_key_iv: "1195d8c8d86bbc5c472a4dd50db5b022"
server_key_version: "encryptionkey@0"
I20260812 11:15:43.567139 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "20a72ba99edd41d0a8c694806493a6b0"
format_stamp: "Formatted at 2026-08-12 11:15:43 on dist-test-slave-zpvx"
server_key: "2dc1ccb358116abadca3e177b9dfb712"
server_key_iv: "1195d8c8d86bbc5c472a4dd50db5b022"
server_key_version: "encryptionkey@0"
I20260812 11:15:43.572139 31436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:15:43.574594 31460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:43.575908 31436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:43.576077 31436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "20a72ba99edd41d0a8c694806493a6b0"
format_stamp: "Formatted at 2026-08-12 11:15:43 on dist-test-slave-zpvx"
server_key: "2dc1ccb358116abadca3e177b9dfb712"
server_key_iv: "1195d8c8d86bbc5c472a4dd50db5b022"
server_key_version: "encryptionkey@0"
I20260812 11:15:43.576233 31436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:43.580878 31436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:43.584023 31436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:43.584245 31436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:43.591346 31436 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35975
I20260812 11:15:43.591396 31547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35975 every 8 connection(s)
I20260812 11:15:43.592243 31436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:43.594969 31549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:43.600333 31549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0: Bootstrap starting.
I20260812 11:15:43.601737 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31436
I20260812 11:15:43.601886 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:43.602166 23943 external_mini_cluster.cc:1428] Setting key 07ebe699723b4090f689cb5d93f59d38
I20260812 11:15:43.603294 31549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:43.604125 31549 log.cc:824] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:43.606225 31549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0: No bootstrap required, opened a new log
I20260812 11:15:43.607549 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:43.603430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:56673 (local address 127.0.0.1:35975)
0812 11:15:43.603901 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:43.603917 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:43.603952 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:43.604869 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:43.604896 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:43.605200 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:43.605569 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:43.606402 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:43.606407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:43.606419 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:43.606430 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:43.606483 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:43.606501 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:43.606623 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:43.606732 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 11:15:43.606895 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":137,"threads_started":1}
I20260812 11:15:43.610263 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:43.610258 31549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260812 11:15:43.610558 31549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:43.610637 31549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a72ba99edd41d0a8c694806493a6b0, State: Initialized, Role: FOLLOWER
I20260812 11:15:43.611079 31549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260812 11:15:43.611238 31549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:43.611310 31549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:43.611409 31549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:43.612315 31549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260812 11:15:43.612749 31549 leader_election.cc:302] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20a72ba99edd41d0a8c694806493a6b0; no voters:
I20260812 11:15:43.613139 31549 leader_election.cc:288] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:43.613240 31569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:43.613476 31569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [term 1 LEADER]: Becoming Leader. State: Replica: 20a72ba99edd41d0a8c694806493a6b0, State: Running, Role: LEADER
I20260812 11:15:43.613802 31569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260812 11:15:43.614300 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:43.612462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:48083 (local address 127.0.0.1:35975)
0812 11:15:43.612565 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:43.612569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:43.612758 (+ 189us) server_negotiation.cc:407] Connection header received
0812 11:15:43.612848 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:43.612854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:43.612889 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:43.612965 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:43.613720 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:43.613724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:43.613726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:43.613738 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:43.613765 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:43.613777 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:43.613871 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:43.614155 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 11:15:43.614194 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 11:15:43.614545 31549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:43.615727 31571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20a72ba99edd41d0a8c694806493a6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } } }
I20260812 11:15:43.615692 31572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20a72ba99edd41d0a8c694806493a6b0. Latest consensus state: current_term: 1 leader_uuid: "20a72ba99edd41d0a8c694806493a6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a72ba99edd41d0a8c694806493a6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } } }
I20260812 11:15:43.615840 31571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:43.615880 31572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:43.616211 31581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:43.619405 31581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:43.625417 31581 catalog_manager.cc:1347] Generated new cluster ID: ab266e1815ac4f6b9c1758737a5673b4
I20260812 11:15:43.625488 31581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:43.644124 31581 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:43.644454 31581 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:43.653638 31581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20a72ba99edd41d0a8c694806493a6b0: Generated new TSK 0
I20260812 11:15:43.654316 31581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:43.661075 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31436
2026-08-12T11:15:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1683 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:43 dist-test-slave-zpvx krb5kdc[31600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:43 dist-test-slave-zpvx krb5kdc[31600](info): set up 2 sockets
Aug 12 11:15:43 dist-test-slave-zpvx krb5kdc[31600](info): commencing operation
krb5kdc: starting...
W20260812 11:15:44.981853 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:45.111471 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:37459
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:45.221132 31626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:45.221419 31626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:45.221495 31626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:45.225376 31626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:45.225469 31626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:45.225510 31626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:45.225543 31626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:45.225572 31626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:45.230616 31626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:37459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:45.231597 31626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:45.232609 31626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:45.241143 31643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:45.241120 31638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:45.241230 31626 server_base.cc:1034] running on GCE node
W20260812 11:15:45.241137 31639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:45.241995 31626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:45.243120 31626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:45.244339 31626 hybrid_clock.cc:648] HybridClock initialized: now 1786533345244319 us; error 31 us; skew 500 ppm
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:45.249903 31626 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:45.250842 31626 webserver.cc:460] Webserver started at http://127.23.97.254:42789/ using document root <none> and password file <none>
I20260812 11:15:45.251420 31626 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:45.251520 31626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:45.251775 31626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:45.253887 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "c199fe92edf541b583e002220e351769"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-zpvx"
server_key: "4c51ef84462e113735497c10d4daebf1"
server_key_iv: "df3fb913332fa7fa22090adc32eced69"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.254475 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "c199fe92edf541b583e002220e351769"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-zpvx"
server_key: "4c51ef84462e113735497c10d4daebf1"
server_key_iv: "df3fb913332fa7fa22090adc32eced69"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.258420 31626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:15:45.260960 31653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:45.262214 31626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:45.262372 31626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "c199fe92edf541b583e002220e351769"
format_stamp: "Formatted at 2026-08-12 11:15:45 on dist-test-slave-zpvx"
server_key: "4c51ef84462e113735497c10d4daebf1"
server_key_iv: "df3fb913332fa7fa22090adc32eced69"
server_key_version: "encryptionkey@0"
I20260812 11:15:45.262487 31626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:45.284430 31626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:45.287416 31626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:45.287616 31626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:45.295029 31626 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:37459
I20260812 11:15:45.295059 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:37459 every 8 connection(s)
I20260812 11:15:45.295985 31626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:45.297662 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31626
I20260812 11:15:45.297775 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:45.298074 23943 external_mini_cluster.cc:1428] Setting key 667bc5ae6c043b1d1f63563afef0c1db
I20260812 11:15:45.298966 31754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:45.304682 31754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769: Bootstrap starting.
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:45.307209 31754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:45.308003 31754 log.cc:824] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:45.310003 31754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769: No bootstrap required, opened a new log
I20260812 11:15:45.312232 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.299523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50183 (local address 127.23.97.254:37459)
0812 11:15:45.299958 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:45.299973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:45.300008 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:15:45.300722 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:45.300740 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:45.301071 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:45.301403 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:45.302238 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.303210 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.304114 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.304650 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.304828 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.307603 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:45.307608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:45.307621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:45.307647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:45.309417 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.309937 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.309943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.309947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.310051 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.310434 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.310437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.310439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.310806 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:45.311032 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:45.311367 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 11:15:45.311578 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260812 11:15:45.313706 31754 raft_consensus.cc:348] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } }
I20260812 11:15:45.313980 31754 raft_consensus.cc:374] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:45.314109 31754 raft_consensus.cc:729] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c199fe92edf541b583e002220e351769, State: Initialized, Role: FOLLOWER
I20260812 11:15:45.314635 31754 consensus_queue.cc:260] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } }
I20260812 11:15:45.314815 31754 raft_consensus.cc:388] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:45.314890 31754 raft_consensus.cc:482] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:45.314990 31754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:45.315353 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:45.316435 31754 raft_consensus.cc:504] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } }
I20260812 11:15:45.316812 31754 leader_election.cc:302] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c199fe92edf541b583e002220e351769; no voters:
I20260812 11:15:45.317159 31754 leader_election.cc:288] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:45.317268 31772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:45.317507 31772 raft_consensus.cc:686] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [term 1 LEADER]: Becoming Leader. State: Replica: c199fe92edf541b583e002220e351769, State: Running, Role: LEADER
I20260812 11:15:45.317951 31772 consensus_queue.cc:237] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } }
I20260812 11:15:45.318452 31754 sys_catalog.cc:564] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:45.319500 31782 sys_catalog.cc:455] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c199fe92edf541b583e002220e351769. Latest consensus state: current_term: 1 leader_uuid: "c199fe92edf541b583e002220e351769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } } }
I20260812 11:15:45.319770 31773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c199fe92edf541b583e002220e351769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c199fe92edf541b583e002220e351769" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 37459 } } }
I20260812 11:15:45.319945 31773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:45.320257 31782 sys_catalog.cc:458] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:45.320427 31801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:45.323891 31801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:45.328064 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:45.317639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56607 (local address 127.23.97.254:37459)
0812 11:15:45.317743 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:45.317747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:45.319234 (+ 1487us) server_negotiation.cc:407] Connection header received
0812 11:15:45.319440 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:45.319444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:45.319491 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:45.319544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:45.320522 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.321258 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.322307 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:45.322631 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:45.322738 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:45.324429 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:45.324434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:45.324436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:45.324461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:45.326216 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.326711 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.326717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.326722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.326809 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:45.327173 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:45.327176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:45.327178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:45.327349 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:45.327510 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:45.327799 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:15:45.327891 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 11:15:45.330541 31801 catalog_manager.cc:1347] Generated new cluster ID: 287d6dd6ba5f4b44bcaad414cdc09ebf
I20260812 11:15:45.330622 31801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:45.341539 31801 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:45.342378 31801 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:45.350867 31801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c199fe92edf541b583e002220e351769: Generated new TSK 0
I20260812 11:15:45.351490 31801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:45.357007 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31626
W20260812 11:15:45.373970 31598 connection.cc:535] client connection to 127.23.97.254:37459 recv error: Network error: failed to read from TLS socket (remote: 127.23.97.254:37459): Connection reset by peer (error 104)
2026-08-12T11:15:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1695 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31819](info): set up 2 sockets
Aug 12 11:15:45 dist-test-slave-zpvx krb5kdc[31819](info): commencing operation
krb5kdc: starting...
W20260812 11:15:46.698212 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:46 dist-test-slave-zpvx krb5kdc[31819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:46.829704 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33397
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:33397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:46.943240 31840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:46.943491 31840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:46.943532 31840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:46.947422 31840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:46.947471 31840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:46.947491 31840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:46.947507 31840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:46.947522 31840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:46.952540 31840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:33397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:46.953451 31840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:46.954388 31840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:46.962831 31850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.962934 31856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:46.962826 31851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:46.963033 31840 server_base.cc:1034] running on GCE node
I20260812 11:15:46.963708 31840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:46.964869 31840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:46.966059 31840 hybrid_clock.cc:648] HybridClock initialized: now 1786533346966026 us; error 49 us; skew 500 ppm
Aug 12 11:15:46 dist-test-slave-zpvx krb5kdc[31819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533346, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:46.968878 31840 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:46.970002 31840 webserver.cc:460] Webserver started at http://127.23.97.254:40475/ using document root <none> and password file <none>
I20260812 11:15:46.970559 31840 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:46.970657 31840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:46.970888 31840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:46.972950 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "0e2310edd08f40c7928c29c0dfc5fca6"
format_stamp: "Formatted at 2026-08-12 11:15:46 on dist-test-slave-zpvx"
server_key: "fda8f488eb55d2d21b9bcfca21ee3fb8"
server_key_iv: "7758b16e5fb8b61282219f1d3b9dd7a6"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.973523 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "0e2310edd08f40c7928c29c0dfc5fca6"
format_stamp: "Formatted at 2026-08-12 11:15:46 on dist-test-slave-zpvx"
server_key: "fda8f488eb55d2d21b9bcfca21ee3fb8"
server_key_iv: "7758b16e5fb8b61282219f1d3b9dd7a6"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.977257 31840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:46.979588 31873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:46.980863 31840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:46.981005 31840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "0e2310edd08f40c7928c29c0dfc5fca6"
format_stamp: "Formatted at 2026-08-12 11:15:46 on dist-test-slave-zpvx"
server_key: "fda8f488eb55d2d21b9bcfca21ee3fb8"
server_key_iv: "7758b16e5fb8b61282219f1d3b9dd7a6"
server_key_version: "encryptionkey@0"
I20260812 11:15:46.981117 31840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:47.017920 31840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:47.021509 31840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:47.021734 31840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:47.029109 31840 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:33397
I20260812 11:15:47.029136 31970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:33397 every 8 connection(s)
I20260812 11:15:47.030062 31840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:47.032809 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:47.036312 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 31840
I20260812 11:15:47.036499 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:47.036827 23943 external_mini_cluster.cc:1428] Setting key d782dea2c17ff8f831b1e5e00bc41592
I20260812 11:15:47.038865 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6: Bootstrap starting.
I20260812 11:15:47.041590 31971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:47.042446 31971 log.cc:824] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:47.044808 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6: No bootstrap required, opened a new log
I20260812 11:15:47.047585 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.038260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37199 (local address 127.23.97.254:33397)
0812 11:15:47.038688 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.038707 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.038761 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:15:47.039558 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.039583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.040032 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.040428 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.040833 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.042318 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.043473 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.044159 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.044331 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.045257 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:47.045264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:47.045280 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:47.045790 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:47.045866 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:47.045897 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:47.046047 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:47.046248 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 11:15:47.046384 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260812 11:15:47.048779 31971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } }
I20260812 11:15:47.049049 31971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:47.049137 31971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2310edd08f40c7928c29c0dfc5fca6, State: Initialized, Role: FOLLOWER
I20260812 11:15:47.049610 31971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } }
I20260812 11:15:47.049775 31971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:47.049872 31971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:47.050000 31971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:47.050843 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:47.051188 31971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } }
I20260812 11:15:47.051568 31971 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2310edd08f40c7928c29c0dfc5fca6; no voters:
I20260812 11:15:47.051923 31971 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:47.051990 31977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:47.052248 31977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [term 1 LEADER]: Becoming Leader. State: Replica: 0e2310edd08f40c7928c29c0dfc5fca6, State: Running, Role: LEADER
I20260812 11:15:47.052623 31977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } }
I20260812 11:15:47.053323 31971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:47.054461 31985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2310edd08f40c7928c29c0dfc5fca6. Latest consensus state: current_term: 1 leader_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } } }
I20260812 11:15:47.054613 31985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:47.054911 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2310edd08f40c7928c29c0dfc5fca6" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33397 } } }
I20260812 11:15:47.054991 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:47.055006 31998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:47.058295 31998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:47.058516 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:47.053155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48731 (local address 127.23.97.254:33397)
0812 11:15:47.053255 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:47.053259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:47.053621 (+ 362us) server_negotiation.cc:407] Connection header received
0812 11:15:47.053744 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:47.053747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:47.053801 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:47.053885 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:47.054316 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.055143 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.056398 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:47.056737 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:47.056842 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:47.058011 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:47.058015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:47.058018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:47.058032 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:47.058062 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:47.058074 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:47.058163 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:47.058349 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 11:15:47.058385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 11:15:47.064759 31998 catalog_manager.cc:1347] Generated new cluster ID: 137cd6ebfb16496ea45950048eec79d8
I20260812 11:15:47.064831 31998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:47.086958 31998 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:47.087925 31998 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:47.099192 31998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2310edd08f40c7928c29c0dfc5fca6: Generated new TSK 0
I20260812 11:15:47.099874 31998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:47.105593 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 31840
W20260812 11:15:47.117785 31817 connection.cc:535] client connection to 127.23.97.254:33397 recv error: Network error: failed to read from TLS socket (remote: 127.23.97.254:33397): Connection reset by peer (error 104)
2026-08-12T11:15:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:47 dist-test-slave-zpvx krb5kdc[32018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:47 dist-test-slave-zpvx krb5kdc[32018](info): set up 2 sockets
Aug 12 11:15:47 dist-test-slave-zpvx krb5kdc[32018](info): commencing operation
krb5kdc: starting...
W20260812 11:15:48.419401 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:48 dist-test-slave-zpvx krb5kdc[32018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:48.555708 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46161
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:48.672850 32039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:48.673192 32039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:48.673283 32039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:48.677328 32039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:48.677410 32039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:48.677448 32039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:48.677505 32039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:48.677541 32039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:48.682575 32039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:48.683652 32039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:48.684767 32039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:15:48.694736 32039 server_base.cc:1034] running on GCE node
W20260812 11:15:48.694672 32047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:48.694639 32048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:48.694644 32052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:48.695529 32039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:48.696614 32039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:48.697803 32039 hybrid_clock.cc:648] HybridClock initialized: now 1786533348697786 us; error 30 us; skew 500 ppm
Aug 12 11:15:48 dist-test-slave-zpvx krb5kdc[32018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533348, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:48.700635 32039 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:48.701417 32039 webserver.cc:460] Webserver started at http://127.23.97.254:46169/ using document root <none> and password file <none>
I20260812 11:15:48.701906 32039 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:48.701962 32039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:48.702121 32039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:48.704028 32039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "aa3585ad3cb8428686224e5e95304fe2"
format_stamp: "Formatted at 2026-08-12 11:15:48 on dist-test-slave-zpvx"
server_key: "ea919d2afb12c6e407b2fee70cd56658"
server_key_iv: "3e4533d8e80bfd8b5f0b4889208770c9"
server_key_version: "encryptionkey@0"
I20260812 11:15:48.704623 32039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "aa3585ad3cb8428686224e5e95304fe2"
format_stamp: "Formatted at 2026-08-12 11:15:48 on dist-test-slave-zpvx"
server_key: "ea919d2afb12c6e407b2fee70cd56658"
server_key_iv: "3e4533d8e80bfd8b5f0b4889208770c9"
server_key_version: "encryptionkey@0"
I20260812 11:15:48.708128 32039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:15:48.710633 32067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:48.711889 32039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:48.712001 32039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "aa3585ad3cb8428686224e5e95304fe2"
format_stamp: "Formatted at 2026-08-12 11:15:48 on dist-test-slave-zpvx"
server_key: "ea919d2afb12c6e407b2fee70cd56658"
server_key_iv: "3e4533d8e80bfd8b5f0b4889208770c9"
server_key_version: "encryptionkey@0"
I20260812 11:15:48.712083 32039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:48.716511 32039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:48.719676 32039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:48.719825 32039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:48.727160 32039 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:46161
I20260812 11:15:48.727246 32168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:46161 every 8 connection(s)
I20260812 11:15:48.728120 32039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:48.730911 32169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:48.733989 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32039
I20260812 11:15:48.734138 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:48.734444 23943 external_mini_cluster.cc:1428] Setting key c0bbb700d138ecce2d98d4cd26ff4c72
I20260812 11:15:48.737186 32169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2: Bootstrap starting.
I20260812 11:15:48.739830 32169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:48.739831 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:48.735713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45203 (local address 127.23.97.254:46161)
0812 11:15:48.736134 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:48.736158 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:48.736204 (+ 46us) server_negotiation.cc:407] Connection header received
0812 11:15:48.737026 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:48.737049 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:48.737398 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:48.737740 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:48.738511 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:48.738519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:48.738534 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:48.738546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:48.738609 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:48.738635 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:48.738768 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:48.738842 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 11:15:48.738981 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260812 11:15:48.740792 32169 log.cc:824] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:48.742627 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:48.742849 32169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2: No bootstrap required, opened a new log
I20260812 11:15:48.746498 32169 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } }
I20260812 11:15:48.746703 32169 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:48.746791 32169 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3585ad3cb8428686224e5e95304fe2, State: Initialized, Role: FOLLOWER
I20260812 11:15:48.747232 32169 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } }
I20260812 11:15:48.747386 32169 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:48.747460 32169 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:48.747601 32169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:48.748381 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:48.744697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46755 (local address 127.23.97.254:46161)
0812 11:15:48.744823 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:48.744827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:48.747003 (+ 2176us) server_negotiation.cc:407] Connection header received
0812 11:15:48.747140 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:48.747142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:48.747178 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:48.747262 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:48.747995 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:48.747997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:48.747999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:48.748010 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:48.748034 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:48.748043 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:48.748118 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:48.748219 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 11:15:48.748248 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:15:48.748843 32169 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } }
I20260812 11:15:48.749217 32169 leader_election.cc:302] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa3585ad3cb8428686224e5e95304fe2; no voters:
I20260812 11:15:48.749554 32169 leader_election.cc:288] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:48.749650 32188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:48.749898 32188 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [term 1 LEADER]: Becoming Leader. State: Replica: aa3585ad3cb8428686224e5e95304fe2, State: Running, Role: LEADER
I20260812 11:15:48.750198 32188 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } }
I20260812 11:15:48.750726 32169 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:48.751856 32190 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3585ad3cb8428686224e5e95304fe2. Latest consensus state: current_term: 1 leader_uuid: "aa3585ad3cb8428686224e5e95304fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } } }
I20260812 11:15:48.751856 32189 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa3585ad3cb8428686224e5e95304fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3585ad3cb8428686224e5e95304fe2" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46161 } } }
I20260812 11:15:48.752034 32190 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:48.752046 32189 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:48.752628 32201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:48.755741 32201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:48.761514 32201 catalog_manager.cc:1347] Generated new cluster ID: d7de0028676f4f16a311955a4f5dd3f8
I20260812 11:15:48.761583 32201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:48.787545 32201 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:48.787873 32201 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:48.795966 32201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa3585ad3cb8428686224e5e95304fe2: Generated new TSK 0
I20260812 11:15:48.796666 32201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:48.803956 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32039
2026-08-12T11:15:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1700 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:48 dist-test-slave-zpvx krb5kdc[32215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:48 dist-test-slave-zpvx krb5kdc[32215](info): set up 2 sockets
Aug 12 11:15:48 dist-test-slave-zpvx krb5kdc[32215](info): commencing operation
krb5kdc: starting...
W20260812 11:15:50.117245 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:50.255715 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40873
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:50.369447 32241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:50.369776 32241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:50.369864 32241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:50.373910 32241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:50.373997 32241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:50.374035 32241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:50.374095 32241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:50.374135 32241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:50.379539 32241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:50.380661 32241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:50.381695 32241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:50.390625 32255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:50.390599 32251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:50.390671 32250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:50.390955 32241 server_base.cc:1034] running on GCE node
I20260812 11:15:50.391521 32241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:50.392717 32241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:50.393923 32241 hybrid_clock.cc:648] HybridClock initialized: now 1786533350393902 us; error 35 us; skew 500 ppm
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533350, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:50.403828 32241 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:50.404850 32241 webserver.cc:460] Webserver started at http://127.23.97.254:36323/ using document root <none> and password file <none>
I20260812 11:15:50.405435 32241 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:50.405532 32241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:50.405772 32241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:50.407847 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "0c868d7ab32940f8aa1cc10747071b7d"
format_stamp: "Formatted at 2026-08-12 11:15:50 on dist-test-slave-zpvx"
server_key: "6ed860b132121123e6f71b052c148bf4"
server_key_iv: "cfa9b5bc575794b2e0bada939c9ca8f6"
server_key_version: "encryptionkey@0"
I20260812 11:15:50.408452 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "0c868d7ab32940f8aa1cc10747071b7d"
format_stamp: "Formatted at 2026-08-12 11:15:50 on dist-test-slave-zpvx"
server_key: "6ed860b132121123e6f71b052c148bf4"
server_key_iv: "cfa9b5bc575794b2e0bada939c9ca8f6"
server_key_version: "encryptionkey@0"
I20260812 11:15:50.412163 32241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:50.414567 32267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:50.415841 32241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:50.416004 32241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "0c868d7ab32940f8aa1cc10747071b7d"
format_stamp: "Formatted at 2026-08-12 11:15:50 on dist-test-slave-zpvx"
server_key: "6ed860b132121123e6f71b052c148bf4"
server_key_iv: "cfa9b5bc575794b2e0bada939c9ca8f6"
server_key_version: "encryptionkey@0"
I20260812 11:15:50.416119 32241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:50.425050 32241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:50.427798 32241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:50.427996 32241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:50.435281 32241 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40873
I20260812 11:15:50.435308 32356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40873 every 8 connection(s)
I20260812 11:15:50.436162 32241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:50.438764 32357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:50.443055 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32241
I20260812 11:15:50.443202 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:50.443527 23943 external_mini_cluster.cc:1428] Setting key 44f24a9b18383b09ccdd312f063ea1de
I20260812 11:15:50.444583 32357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d: Bootstrap starting.
I20260812 11:15:50.446826 32357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:50.447556 32357 log.cc:824] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:50.449492 32357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d: No bootstrap required, opened a new log
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:50.453902 32357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } }
I20260812 11:15:50.454151 32357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:50.454213 32357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c868d7ab32940f8aa1cc10747071b7d, State: Initialized, Role: FOLLOWER
I20260812 11:15:50.454689 32357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } }
I20260812 11:15:50.454843 32357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:50.454916 32357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:50.455031 32357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:50.455876 32357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } }
I20260812 11:15:50.456247 32357 leader_election.cc:302] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c868d7ab32940f8aa1cc10747071b7d; no voters:
I20260812 11:15:50.456617 32357 leader_election.cc:288] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:50.456724 32363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:50.456992 32363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [term 1 LEADER]: Becoming Leader. State: Replica: 0c868d7ab32940f8aa1cc10747071b7d, State: Running, Role: LEADER
I20260812 11:15:50.457362 32363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } }
I20260812 11:15:50.457996 32357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:50.458638 32363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c868d7ab32940f8aa1cc10747071b7d. Latest consensus state: current_term: 1 leader_uuid: "0c868d7ab32940f8aa1cc10747071b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } } }
I20260812 11:15:50.458766 32363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:50.459071 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:50.444928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39865 (local address 127.23.97.254:40873)
0812 11:15:50.445342 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:50.445357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:50.445393 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:15:50.446007 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:50.446023 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:50.446346 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:50.446696 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:50.447533 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.448809 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.449713 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.450379 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.450555 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.453422 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:50.453428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:50.453444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:50.453475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:50.455618 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:50.456089 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:50.456096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:50.456101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:50.456258 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:50.456699 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:50.456704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:50.456706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:50.457269 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:50.457453 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:50.457793 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 11:15:50.458088 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260812 11:15:50.459219 32364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c868d7ab32940f8aa1cc10747071b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c868d7ab32940f8aa1cc10747071b7d" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40873 } } }
I20260812 11:15:50.459316 32364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:50.460214 32374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:50.462237 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:50.463464 32374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:50.469045 32374 catalog_manager.cc:1347] Generated new cluster ID: 4d9b8b5e182e41e0bfb032ce34b563da
I20260812 11:15:50.469147 32374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:50.473968 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:50.464523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52125 (local address 127.23.97.254:40873)
0812 11:15:50.464627 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:50.464637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:50.464967 (+ 330us) server_negotiation.cc:407] Connection header received
0812 11:15:50.465098 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:50.465102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:50.465151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:50.465233 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:50.466078 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.466882 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.468498 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:50.468834 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:50.468947 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:50.470615 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:50.470619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:50.470622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:50.470655 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:50.472165 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:50.472702 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:50.472706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:50.472710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:50.472792 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:50.473089 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:50.473092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:50.473094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:50.473276 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:50.473419 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:50.473669 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 11:15:50.473786 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:15:50.489236 32374 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:50.490489 32374 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:50.498309 32374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c868d7ab32940f8aa1cc10747071b7d: Generated new TSK 0
I20260812 11:15:50.499135 32374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:50.507651 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32241
2026-08-12T11:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1704 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32411](info): set up 2 sockets
Aug 12 11:15:50 dist-test-slave-zpvx krb5kdc[32411](info): commencing operation
krb5kdc: starting...
W20260812 11:15:51.838802 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:51 dist-test-slave-zpvx krb5kdc[32411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:51.967741 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33221
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:52.080500 32437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:52.080804 32437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:52.080880 32437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:52.085186 32437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:52.085304 32437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:52.085345 32437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:52.085409 32437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:52.085449 32437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:52.090656 32437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:52.091688 32437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:52.092720 32437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:52.101209 32446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:52.101238 32447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:52.101305 32437 server_base.cc:1034] running on GCE node
W20260812 11:15:52.101248 32451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:52.102008 32437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:52.103050 32437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:52.104270 32437 hybrid_clock.cc:648] HybridClock initialized: now 1786533352104240 us; error 42 us; skew 500 ppm
Aug 12 11:15:52 dist-test-slave-zpvx krb5kdc[32411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533352, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:52.107275 32437 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:52.108206 32437 webserver.cc:460] Webserver started at http://127.23.97.254:36731/ using document root <none> and password file <none>
I20260812 11:15:52.108829 32437 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:52.108923 32437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:52.109185 32437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:52.111234 32437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "72360d0cb39449be9c36790cab4b6cfc"
format_stamp: "Formatted at 2026-08-12 11:15:52 on dist-test-slave-zpvx"
server_key: "de3cd919195b42fffa67761ec5195bec"
server_key_iv: "82b42e77dfcf822f01040ddeafac4c2d"
server_key_version: "encryptionkey@0"
I20260812 11:15:52.111809 32437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "72360d0cb39449be9c36790cab4b6cfc"
format_stamp: "Formatted at 2026-08-12 11:15:52 on dist-test-slave-zpvx"
server_key: "de3cd919195b42fffa67761ec5195bec"
server_key_iv: "82b42e77dfcf822f01040ddeafac4c2d"
server_key_version: "encryptionkey@0"
I20260812 11:15:52.115677 32437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:15:52.118204 32468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:52.119388 32437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:15:52.119545 32437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "72360d0cb39449be9c36790cab4b6cfc"
format_stamp: "Formatted at 2026-08-12 11:15:52 on dist-test-slave-zpvx"
server_key: "de3cd919195b42fffa67761ec5195bec"
server_key_iv: "82b42e77dfcf822f01040ddeafac4c2d"
server_key_version: "encryptionkey@0"
I20260812 11:15:52.119657 32437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:52.134254 32437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:52.137231 32437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:52.137436 32437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:52.144559 32437 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:33221
I20260812 11:15:52.144588 32564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:33221 every 8 connection(s)
I20260812 11:15:52.145517 32437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:52.148394 32565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:52.154435 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32437
I20260812 11:15:52.154529 32565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc: Bootstrap starting.
I20260812 11:15:52.154600 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:52.154939 23943 external_mini_cluster.cc:1428] Setting key f416f333337168d5d04d5c34ef3371c6
I20260812 11:15:52.157348 32565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:52.158052 32565 log.cc:824] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:52.160223 32565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc: No bootstrap required, opened a new log
I20260812 11:15:52.164603 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:52.156446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58161 (local address 127.23.97.254:33221)
0812 11:15:52.156856 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:52.156873 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:52.156909 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:15:52.157607 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:52.157626 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:52.157963 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:52.158305 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:52.158723 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.159730 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.161162 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.161642 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.161775 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.162714 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:52.162720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:52.162733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:52.163157 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:52.163217 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:52.163241 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:52.163325 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:52.163539 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 11:15:52.163759 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260812 11:15:52.164775 32565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } }
I20260812 11:15:52.165081 32565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:52.165148 32565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72360d0cb39449be9c36790cab4b6cfc, State: Initialized, Role: FOLLOWER
I20260812 11:15:52.165620 32565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } }
I20260812 11:15:52.165796 32565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:52.165882 32565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:52.166014 32565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:52.167116 32565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } }
I20260812 11:15:52.167506 32565 leader_election.cc:302] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72360d0cb39449be9c36790cab4b6cfc; no voters:
I20260812 11:15:52.167745 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:52.167896 32565 leader_election.cc:288] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:52.168275 32576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:52.168542 32576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [term 1 LEADER]: Becoming Leader. State: Replica: 72360d0cb39449be9c36790cab4b6cfc, State: Running, Role: LEADER
I20260812 11:15:52.168932 32576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } }
I20260812 11:15:52.169265 32565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:52.170389 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72360d0cb39449be9c36790cab4b6cfc. Latest consensus state: current_term: 1 leader_uuid: "72360d0cb39449be9c36790cab4b6cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } } }
I20260812 11:15:52.170620 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:52.170694 32579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72360d0cb39449be9c36790cab4b6cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72360d0cb39449be9c36790cab4b6cfc" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 33221 } } }
I20260812 11:15:52.170774 32579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:52.171085 32595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:52.173906 32595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:52.179747 32595 catalog_manager.cc:1347] Generated new cluster ID: 942a4a5b43734dc884f8786b2ec81d3f
I20260812 11:15:52.179874 32595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:52.180020 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:52.170129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60821 (local address 127.23.97.254:33221)
0812 11:15:52.170253 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:52.170259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:52.175785 (+ 5526us) server_negotiation.cc:407] Connection header received
0812 11:15:52.175963 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:52.175968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:52.176024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:52.176105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:52.176529 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.177057 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.177886 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:52.178210 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:52.178334 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:52.179271 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:52.179275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:52.179278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:52.179292 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:52.179322 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:52.179336 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:52.179417 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:52.179746 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 11:15:52.179805 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:15:52.192576 32595 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:52.193795 32595 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:52.205739 32595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72360d0cb39449be9c36790cab4b6cfc: Generated new TSK 0
I20260812 11:15:52.206434 32595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:52.217041 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32437
2026-08-12T11:15:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1709 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:52 dist-test-slave-zpvx krb5kdc[32615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:52 dist-test-slave-zpvx krb5kdc[32615](info): set up 2 sockets
Aug 12 11:15:52 dist-test-slave-zpvx krb5kdc[32615](info): commencing operation
krb5kdc: starting...
W20260812 11:15:53.574910 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:53 dist-test-slave-zpvx krb5kdc[32615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:53.706852 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40131
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:40131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:53.818117 32633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:53.818394 32633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:53.818434 32633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:53.822237 32633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:53.822283 32633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:53.822301 32633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:53.822317 32633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:53.822332 32633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:53.827261 32633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:40131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:40131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:53.828162 32633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:53.829119 32633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:53.837421 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:53.837466 32633 server_base.cc:1034] running on GCE node
W20260812 11:15:53.837419 32640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:53.837419 32648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:53.838172 32633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:53.839145 32633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:53.840344 32633 hybrid_clock.cc:648] HybridClock initialized: now 1786533353840325 us; error 31 us; skew 500 ppm
Aug 12 11:15:53 dist-test-slave-zpvx krb5kdc[32615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533353, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:53.843122 32633 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:53.843956 32633 webserver.cc:460] Webserver started at http://127.23.97.254:40123/ using document root <none> and password file <none>
I20260812 11:15:53.844570 32633 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:53.844663 32633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:53.844901 32633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:53.846863 32633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "160f56f44ee84a76b43048f737037a06"
format_stamp: "Formatted at 2026-08-12 11:15:53 on dist-test-slave-zpvx"
server_key: "0c36e2aa01fc44934a28b960f7bab742"
server_key_iv: "e6714c10f55fdc9d75437ea8f65a3359"
server_key_version: "encryptionkey@0"
I20260812 11:15:53.847429 32633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "160f56f44ee84a76b43048f737037a06"
format_stamp: "Formatted at 2026-08-12 11:15:53 on dist-test-slave-zpvx"
server_key: "0c36e2aa01fc44934a28b960f7bab742"
server_key_iv: "e6714c10f55fdc9d75437ea8f65a3359"
server_key_version: "encryptionkey@0"
I20260812 11:15:53.851071 32633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 11:15:53.853485 32657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:53.854619 32633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:53.854779 32633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "160f56f44ee84a76b43048f737037a06"
format_stamp: "Formatted at 2026-08-12 11:15:53 on dist-test-slave-zpvx"
server_key: "0c36e2aa01fc44934a28b960f7bab742"
server_key_iv: "e6714c10f55fdc9d75437ea8f65a3359"
server_key_version: "encryptionkey@0"
I20260812 11:15:53.854892 32633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:53.859319 32633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:53.862066 32633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:53.862258 32633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:53.869109 32633 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:40131
I20260812 11:15:53.869140 32748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:40131 every 8 connection(s)
I20260812 11:15:53.869989 32633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:53.872664 32749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:53.872712 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 32633
I20260812 11:15:53.872869 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:53.873155 23943 external_mini_cluster.cc:1428] Setting key 261cc8802bd66eb96002934add909d68
I20260812 11:15:53.878341 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:53.874501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53539 (local address 127.23.97.254:40131)
0812 11:15:53.874943 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:53.874957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:53.874991 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:15:53.875794 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:53.875813 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:53.876082 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:53.876466 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:53.877193 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:53.877198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:53.877219 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:53.877231 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:53.877286 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:53.877306 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:53.877462 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:53.877532 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 11:15:53.877676 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260812 11:15:53.879345 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06: Bootstrap starting.
I20260812 11:15:53.881167 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:53.881867 32749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:53.882647 32749 log.cc:824] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:53.884893 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:53.883250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35923 (local address 127.23.97.254:40131)
0812 11:15:53.883360 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:53.883365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:53.883587 (+ 222us) server_negotiation.cc:407] Connection header received
0812 11:15:53.883707 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:53.883709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:53.883737 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:53.883804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:53.884486 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:53.884489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:53.884491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:15:53.884501 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:53.884525 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:15:53.884535 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:53.884604 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:53.884725 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 11:15:53.884759 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 11:15:53.885046 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06: No bootstrap required, opened a new log
I20260812 11:15:53.888356 32749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } }
I20260812 11:15:53.888633 32749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:15:53.888710 32749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 160f56f44ee84a76b43048f737037a06, State: Initialized, Role: FOLLOWER
I20260812 11:15:53.889160 32749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } }
I20260812 11:15:53.889293 32749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:15:53.889417 32749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:15:53.889542 32749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:53.890421 32749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } }
I20260812 11:15:53.890763 32749 leader_election.cc:302] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 160f56f44ee84a76b43048f737037a06; no voters:
I20260812 11:15:53.891065 32749 leader_election.cc:288] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:15:53.891196 32764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:53.891435 32764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [term 1 LEADER]: Becoming Leader. State: Replica: 160f56f44ee84a76b43048f737037a06, State: Running, Role: LEADER
I20260812 11:15:53.891805 32764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } }
I20260812 11:15:53.892256 32749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:53.893465 32767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 160f56f44ee84a76b43048f737037a06. Latest consensus state: current_term: 1 leader_uuid: "160f56f44ee84a76b43048f737037a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } } }
I20260812 11:15:53.893599 32767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:53.894021 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "160f56f44ee84a76b43048f737037a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160f56f44ee84a76b43048f737037a06" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 40131 } } }
I20260812 11:15:53.894040 312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:53.894649 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06 [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:53.897372 312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:53.902210 312 catalog_manager.cc:1347] Generated new cluster ID: 4a0602ab6acb40769daa392c60c28f04
I20260812 11:15:53.902274 312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:53.919478 312 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:53.919795 312 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:53.933601 312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 160f56f44ee84a76b43048f737037a06: Generated new TSK 0
I20260812 11:15:53.934317 312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:15:53.945312 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 32633
2026-08-12T11:15:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1733 ms)
[----------] 18 tests from AuthTokenIssuingTest (31996 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:54 dist-test-slave-zpvx krb5kdc[340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:54 dist-test-slave-zpvx krb5kdc[340](info): set up 2 sockets
Aug 12 11:15:54 dist-test-slave-zpvx krb5kdc[340](info): commencing operation
krb5kdc: starting...
W20260812 11:15:55.265981 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:55.390581 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34021
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:55.500311 364 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:55.500589 364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:55.500613 364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:55.500650 364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:55.504456 364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:55.504499 364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:55.504518 364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:55.504534 364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:55.504550 364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:55.509500 364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:55.510386 364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:55.511299 364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:55.519642 371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:55.519637 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.519846 364 server_base.cc:1034] running on GCE node
W20260812 11:15:55.519682 375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.520599 364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:55.521566 364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:55.522729 364 hybrid_clock.cc:648] HybridClock initialized: now 1786533355522713 us; error 32 us; skew 500 ppm
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:55.530669 364 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:55.531483 364 webserver.cc:460] Webserver started at http://127.23.97.254:36093/ using document root <none> and password file <none>
I20260812 11:15:55.531988 364 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:55.532053 364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:55.532222 364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:55.534250 364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "5077bc9033c94db1b29af5c8d1826ebe"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "39b564ac5ceeaee3ddd83dd1417557c1"
server_key_iv: "5de7150a0613673a505265fddef61147"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.534755 364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "5077bc9033c94db1b29af5c8d1826ebe"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "39b564ac5ceeaee3ddd83dd1417557c1"
server_key_iv: "5de7150a0613673a505265fddef61147"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.538391 364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:15:55.540755 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:55.542048 364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:15:55.542222 364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "5077bc9033c94db1b29af5c8d1826ebe"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "39b564ac5ceeaee3ddd83dd1417557c1"
server_key_iv: "5de7150a0613673a505265fddef61147"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.542337 364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:55.555158 364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:55.558383 364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:55.558586 364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:55.565869 364 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:34021
I20260812 11:15:55.566202 481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:34021 every 8 connection(s)
I20260812 11:15:55.566820 364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:55.569695 484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:55.573642 484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:55.576756 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 364
I20260812 11:15:55.576922 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:55.577243 23943 external_mini_cluster.cc:1428] Setting key 139f4e8676c484c9f7f217fb6b5f7deb
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.253@KRBTEST.COM" created.
I20260812 11:15:55.590600 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.575745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.23.97.254:34021)
0812 11:15:55.576198 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.576215 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.576414 (+ 199us) server_negotiation.cc:407] Connection header received
0812 11:15:55.577882 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.577887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.578062 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.578216 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.579467 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.580928 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.582652 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.583010 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.583201 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.585561 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:55.585575 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:55.585585 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:55.585605 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:55.587336 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.587864 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.587877 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.587878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.587977 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.588299 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.588304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.588306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.588826 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:55.588972 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:55.589360 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 11:15:55.589713 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":336,"spinlock_wait_cycles":28032,"thread_start_us":179,"threads_started":1}
I20260812 11:15:55.590600 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.575745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34021 (local address 127.0.0.1:56820)
0812 11:15:55.576255 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.576297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.577565 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.578292 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.578301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.578820 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.579249 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.579272 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.581083 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.581091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.582499 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.582515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.583216 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.583219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.583350 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.583913 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.583930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.585414 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:55.587449 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.587460 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.587469 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.587750 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.588075 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.588080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.588082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.588204 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.588947 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:55.588953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:55.589106 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 11:15:55.589281 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 11:15:55.589609 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260812 11:15:55.593032 484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:55.593758 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.593505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:44980)
0812 11:15:55.593619 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.593702 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:15:55.593984 397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
I20260812 11:15:55.594806 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.594598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:44988)
0812 11:15:55.594705 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.594757 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:15:55.594970 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36747: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
I20260812 11:15:55.605645 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:36747
--webserver_interface=127.23.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:55.644212 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } attempt: 1
I20260812 11:15:55.645192 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.644810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:44992)
0812 11:15:55.644991 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.645095 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 11:15:55.645942 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.645715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45000)
0812 11:15:55.645825 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.645879 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 11:15:55.646097 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36747: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
I20260812 11:15:55.714269 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } attempt: 2
I20260812 11:15:55.715188 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.714857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45008)
0812 11:15:55.715021 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.715091 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:15:55.715970 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.715758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45018)
0812 11:15:55.715864 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.715914 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:15:55.716184 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:36747: Network error: Client connection negotiation failed: client connection to 127.23.97.253:36747: connect: Connection refused (error 111)
W20260812 11:15:55.717617 495 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:55.717868 495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:55.717931 495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:55.718009 495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:55.721982 495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:55.722029 495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:55.722048 495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:55.722065 495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:55.722126 495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:55.727182 495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:55.728174 495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:55.729229 495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:55.737908 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.737998 495 server_base.cc:1034] running on GCE node
W20260812 11:15:55.737932 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:55.737910 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.738819 495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:55.739879 495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:55.741051 495 hybrid_clock.cc:648] HybridClock initialized: now 1786533355741024 us; error 37 us; skew 500 ppm
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:55.743844 495 init.cc:377] Logged in from keytab as kudu/127.23.97.253@KRBTEST.COM (short username kudu)
I20260812 11:15:55.744781 495 webserver.cc:460] Webserver started at http://127.23.97.253:33595/ using document root <none> and password file <none>
I20260812 11:15:55.745337 495 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:55.745395 495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:55.745613 495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:55.747720 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data/instance:
uuid: "a9a7800b83504c39b8b04be434453406"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "06330a87d24b8db1f92814ee2bcb6958"
server_key_iv: "2702189c320924ffb27088c5c68cc450"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.748296 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal/instance:
uuid: "a9a7800b83504c39b8b04be434453406"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "06330a87d24b8db1f92814ee2bcb6958"
server_key_iv: "2702189c320924ffb27088c5c68cc450"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.751959 495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:55.754449 532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:55.755703 495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:55.755844 495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "a9a7800b83504c39b8b04be434453406"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "06330a87d24b8db1f92814ee2bcb6958"
server_key_iv: "2702189c320924ffb27088c5c68cc450"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.755959 495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:55.776525 495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:55.779628 495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:55.779834 495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:55.787177 495 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.253:36747
I20260812 11:15:55.787552 620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.253:36747 every 8 connection(s)
I20260812 11:15:55.788436 495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
I20260812 11:15:55.791695 624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:55.791906 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 495
I20260812 11:15:55.792057 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-1/wal/instance
I20260812 11:15:55.792414 23943 external_mini_cluster.cc:1428] Setting key 2c1920adf861a79bd3023ec401e14372
I20260812 11:15:55.795634 624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.97.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.252@KRBTEST.COM" created.
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:55.807339 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } attempt: 3
I20260812 11:15:55.811066 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.797593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.23.97.254:34021)
0812 11:15:55.797756 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.797762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.798635 (+ 873us) server_negotiation.cc:407] Connection header received
0812 11:15:55.799564 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.799568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.799636 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.799763 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.800987 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.801809 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.803089 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.803461 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.803567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.806088 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:55.806091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:55.806093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:55.806116 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:55.807852 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.808579 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.808583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.808584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.808679 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.809087 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.809091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.809093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.809349 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:55.809470 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:55.810785 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0812 11:15:55.810908 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:55.811661 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.797982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34021 (local address 127.0.0.1:56836)
0812 11:15:55.798449 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.798500 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.799344 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.799892 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.799899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.800262 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.800823 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.800837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.801953 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.801961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.802955 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.802967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.803620 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.803623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.803770 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.804477 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.804494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.805909 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:55.807998 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.808007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.808026 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.808478 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.808791 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.808794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.808796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.808963 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.809468 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:55.809475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:55.809756 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 11:15:55.809969 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 11:15:55.810675 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":267,"thread_start_us":140,"threads_started":1}
I20260812 11:15:55.813167 624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
I20260812 11:15:55.821341 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.807804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45020)
0812 11:15:55.808002 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.808019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.808103 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.808809 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.808816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.809031 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.809212 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.809223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.810829 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.810834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.811789 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.811803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.812741 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.812744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.812847 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.813522 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.813535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.816906 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:55.819460 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.819463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.819464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.819662 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.820031 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.820033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.820035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.820113 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.820672 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:55.820675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:55.820803 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 11:15:55.820928 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 11:15:55.821089 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 11:15:55.821455 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.807910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.23.97.253:36747)
0812 11:15:55.808327 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.808336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.808355 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:55.808431 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.808436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.808647 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.808795 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.809371 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.810664 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.811939 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.812585 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.812754 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.817096 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:55.817105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:55.817115 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:55.817135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:55.819299 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.819796 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.819804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.819807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.819931 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.820217 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.820223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.820225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.820570 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:55.820713 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:55.821039 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 11:15:55.821251 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":134,"threads_started":1}
I20260812 11:15:55.821954 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:33247
--webserver_interface=127.23.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:55.822916 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.813714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45022)
0812 11:15:55.813885 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.813898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:55.813968 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:55.814316 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:55.814318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:55.814481 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:55.814555 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.814560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.815345 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.815352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.815844 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:55.815851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.816476 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.816480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:55.816611 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.817126 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:55.817157 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:55.818544 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:55.821484 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.821489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.821492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.821725 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.821981 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:55.821983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:55.821984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:55.822053 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:55.822436 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:55.822439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:55.822521 (+ 82us) client_negotiation.cc:769] Sending connection context
0812 11:15:55.822624 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:15:55.822757 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"spinlock_wait_cycles":768}
I20260812 11:15:55.822973 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.813760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.23.97.253:36747)
0812 11:15:55.814107 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:55.814116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:55.814129 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:55.814161 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:55.814163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:55.814206 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:55.814306 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:55.814668 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.815242 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.815957 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:55.816320 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:55.816481 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:55.818751 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:55.818755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:55.818758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:55.818783 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:55.821368 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.821826 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.821829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.821831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.821901 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:55.822135 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:55.822140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:55.822143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:55.822331 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:55.822432 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:55.822668 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 11:15:55.822815 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260812 11:15:55.823510 624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:55.823503 484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:55.824244 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.823987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43380)
0812 11:15:55.824151 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.824194 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 11:15:55.824282 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.823917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43370)
0812 11:15:55.824164 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.824234 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
W20260812 11:15:55.824461 546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
I20260812 11:15:55.825214 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.824897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43388)
0812 11:15:55.825081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.825134 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 11:15:55.825275 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.825051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43394)
0812 11:15:55.825160 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.825217 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 11:15:55.825431 624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
W20260812 11:15:55.825614 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
I20260812 11:15:55.856804 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 1
I20260812 11:15:55.857751 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.857384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43406)
0812 11:15:55.857561 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.857641 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 11:15:55.858546 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.858298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43410)
0812 11:15:55.858415 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.858473 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:15:55.858697 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
I20260812 11:15:55.874692 624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 1
I20260812 11:15:55.875590 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.875243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43420)
0812 11:15:55.875417 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.875491 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:15:55.876359 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.876125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43432)
0812 11:15:55.876238 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.876292 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:15:55.876524 624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
I20260812 11:15:55.933873 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 2
I20260812 11:15:55.934769 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.934416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43444)
0812 11:15:55.934591 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.934661 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:15:55.935508 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.935282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43452)
0812 11:15:55.935397 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.935450 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:15:55.935665 484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
I20260812 11:15:55.944684 624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 2
W20260812 11:15:55.944805 635 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:55.945160 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:55.945228 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:55.945310 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 11:15:55.945497 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.945195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43458)
0812 11:15:55.945360 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.945422 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 11:15:55.946239 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:55.946007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43462)
0812 11:15:55.946129 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:55.946177 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 11:15:55.946376 624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:33247: Network error: Client connection negotiation failed: client connection to 127.23.97.252:33247: connect: Connection refused (error 111)
W20260812 11:15:55.949712 635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:55.949790 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:55.949836 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:55.949872 635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:55.949905 635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:55.955624 635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:34021,127.23.97.253:36747,127.23.97.252:33247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:55.956750 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:55.957753 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:55.966377 643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:55.966393 652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.966470 635 server_base.cc:1034] running on GCE node
W20260812 11:15:55.966382 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:55.967202 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:55.968232 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:55.969401 635 hybrid_clock.cc:648] HybridClock initialized: now 1786533355969373 us; error 48 us; skew 500 ppm
Aug 12 11:15:55 dist-test-slave-zpvx krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:55.972304 635 init.cc:377] Logged in from keytab as kudu/127.23.97.252@KRBTEST.COM (short username kudu)
I20260812 11:15:55.973243 635 webserver.cc:460] Webserver started at http://127.23.97.252:45965/ using document root <none> and password file <none>
I20260812 11:15:55.973768 635 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:55.973826 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:55.973990 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:55.975950 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data/instance:
uuid: "a8aea5bd77df4ac1a5b2948633f48537"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "1a5357bea8f9e7d655cb52c408fd873c"
server_key_iv: "f420e5605d37f83ae09018fed6038e66"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.976543 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal/instance:
uuid: "a8aea5bd77df4ac1a5b2948633f48537"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "1a5357bea8f9e7d655cb52c408fd873c"
server_key_iv: "f420e5605d37f83ae09018fed6038e66"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.980223 635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:15:55.982681 670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:55.983992 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:55.984201 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "a8aea5bd77df4ac1a5b2948633f48537"
format_stamp: "Formatted at 2026-08-12 11:15:55 on dist-test-slave-zpvx"
server_key: "1a5357bea8f9e7d655cb52c408fd873c"
server_key_iv: "f420e5605d37f83ae09018fed6038e66"
server_key_version: "encryptionkey@0"
I20260812 11:15:55.984306 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:55.996255 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:55.999511 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:55.999723 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:56.006856 635 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.252:33247
I20260812 11:15:56.007190 752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.252:33247 every 8 connection(s)
I20260812 11:15:56.007941 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
I20260812 11:15:56.010056 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 635
I20260812 11:15:56.010208 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533060973827-23943-0/minicluster-data/master-2/wal/instance
I20260812 11:15:56.010524 23943 external_mini_cluster.cc:1428] Setting key 30797d9482d3cdfc7fe178ee22d7ad16
I20260812 11:15:56.011062 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:56.015444 753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:56.026135 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.011977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.23.97.252:33247)
0812 11:15:56.012475 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.012491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.012528 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:15:56.013251 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.013270 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.013602 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.013943 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.014794 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.015781 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.017070 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.017737 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.017927 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.020918 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.020923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.020933 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.020960 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.022794 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.023318 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.023322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.023327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.023469 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.023779 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.023785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.023789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.024310 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.024547 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.024886 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.025175 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260812 11:15:56.027006 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.017340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:34021 (local address 127.0.0.1:56848)
0812 11:15:56.017773 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.017827 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.017980 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.018381 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.018388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.018812 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.018947 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.018953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.019965 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.019971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.020797 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.020807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.021265 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.021269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.021389 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.021910 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.021922 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.023345 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:56.025383 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.025392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.025396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.025688 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.025989 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.025991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.025992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.026066 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.026526 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:56.026531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:56.026636 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 11:15:56.026760 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:15:56.026885 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":149,"threads_started":1}
I20260812 11:15:56.027364 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.017406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.23.97.254:34021)
0812 11:15:56.017558 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.017563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.017896 (+ 333us) server_negotiation.cc:407] Connection header received
0812 11:15:56.018107 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.018113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.018219 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.018298 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.019084 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.019833 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.020942 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.021146 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.021263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.023489 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.023495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.023502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.023537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.025271 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.025795 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.025799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.025801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.025894 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.026171 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.026175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.026177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.026417 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.026521 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.026843 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.026960 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:15:56.028313 753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:56.034821 484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 3
I20260812 11:15:56.037304 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.028812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:36747 (local address 127.0.0.1:45032)
0812 11:15:56.028936 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.028949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.029016 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.029557 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.029559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.029695 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.029778 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.029782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.030569 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.030571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.031056 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.031063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.031532 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.031538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.031648 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.032038 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.032049 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.033436 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:56.035474 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.035479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.035481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.035889 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.036186 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.036189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.036191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.036285 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.036799 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:56.036802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:56.036910 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 11:15:56.037023 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:15:56.037149 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:15:56.037364 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.029141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.23.97.253:36747)
0812 11:15:56.029279 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.029283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.029321 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:15:56.029371 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.029375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.029433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.029523 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.029933 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.030454 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.031178 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.031423 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.031504 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.033577 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.033580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.033583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.033606 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.035346 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.035976 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.035980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.035982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.036074 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.036424 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.036430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.036435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.036685 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.036789 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.037090 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.037210 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:15:56.037539 624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } attempt: 3
I20260812 11:15:56.038451 753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:56.038919 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.028714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.23.97.254:34021)
0812 11:15:56.028826 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.028829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.028859 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:15:56.028968 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.028974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.029018 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.029109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.029878 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.030380 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.031228 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.031505 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.031611 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.034229 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.034233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.034235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.034260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.035977 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.036489 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.036493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.036494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.036594 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.036919 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.036923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.036924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.037143 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.037367 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.038633 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.038758 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
I20260812 11:15:56.045146 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.035669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43480)
0812 11:15:56.035843 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.035857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.035933 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.036195 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.036198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.036401 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.036501 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.036507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.037414 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.037417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.038177 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.038193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.039048 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.039052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.039150 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.039616 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.039639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.041197 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:56.043169 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.043172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.043173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.043441 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.043819 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.043825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.043827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.043919 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.044599 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:56.044603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:56.044704 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 11:15:56.044816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 11:15:56.044954 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 11:15:56.045571 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.035742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.23.97.252:33247)
0812 11:15:56.035870 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.035875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.035893 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:56.036049 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.036054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.036098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.036182 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.036620 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.037277 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.038574 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.038914 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.039537 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.041378 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.041381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.041384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.041400 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.043051 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.043575 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.043581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.043585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.043698 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.044045 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.044048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.044049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.044472 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.045212 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.045302 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.045418 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":46}
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:15:56 dist-test-slave-zpvx krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:15:56.049338 484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: Bootstrap starting.
I20260812 11:15:56.050340 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.037900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43482)
0812 11:15:56.038349 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.038362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.038429 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.038739 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.038743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.038897 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.039001 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.039011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.041047 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.041050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.042446 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.042462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.043167 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.043171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.043268 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.043726 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.043746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.045354 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:56.048740 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.048743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.048745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.049004 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.049363 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.049366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.049368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.049462 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.049904 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:56.049907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:56.049998 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 11:15:56.050077 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 11:15:56.050199 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377}
I20260812 11:15:56.050886 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.037960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.23.97.252:33247)
0812 11:15:56.038404 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.038413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.038431 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:15:56.038508 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.038511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.038555 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.039608 (+ 1053us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.039719 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.040949 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.042602 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.043009 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.044404 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.046835 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.046838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.046840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.046861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.048600 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.049155 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.049158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.049159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.049249 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.049583 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.049585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.049587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.049788 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.050502 (+ 714us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.050584 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.050723 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"spinlock_wait_cycles":768,"thread_start_us":96,"threads_started":1}
I20260812 11:15:56.051792 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.038866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:33247 (local address 127.0.0.1:43496)
0812 11:15:56.039156 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:56.039169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:56.039236 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:56.041418 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:56.041421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:56.041607 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:56.041691 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.041697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.042701 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.042704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.043483 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:56.043493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.044070 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.044073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:56.044141 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.044664 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:56.044676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:56.046464 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:56.049724 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.049728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.049730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.049959 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.050264 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:56.050269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:56.050272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:56.050358 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:56.051334 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:56.051337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:56.051422 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 11:15:56.051505 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:15:56.051634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":27}
I20260812 11:15:56.052105 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.040444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.23.97.253:36747)
0812 11:15:56.040568 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.040572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.040625 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:15:56.040728 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.040730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.040772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.040855 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.041634 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.042380 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.043477 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.043795 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.043882 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.048047 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.048052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.048055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.048083 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.049919 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.050426 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.050429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.050432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.050528 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.050860 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.050865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.050866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.051069 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.051276 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.051865 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.051974 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 11:15:56.053109 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.039066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.23.97.252:33247)
0812 11:15:56.041189 (+ 2123us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.041199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.041212 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:56.041249 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.041253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.041300 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.041785 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.041889 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.042599 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.043606 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.043963 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.044200 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.047163 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.047168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.047171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.047200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.049604 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.050062 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.050079 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.050084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.050161 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.050801 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.050806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.050811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.051230 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.052779 (+ 1549us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.052849 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.052950 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":99,"threads_started":1}
I20260812 11:15:56.053632 484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:56.054416 624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: Bootstrap starting.
I20260812 11:15:56.054472 484 log.cc:824] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:56.055634 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:56.056811 484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: No bootstrap required, opened a new log
I20260812 11:15:56.057039 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: Bootstrap starting.
I20260812 11:15:56.057605 624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:56.057096 440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5077bc9033c94db1b29af5c8d1826ebe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56852
I20260812 11:15:56.058482 624 log.cc:824] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:56.059212 753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:56.060046 753 log.cc:824] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:56.060783 624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: No bootstrap required, opened a new log
I20260812 11:15:56.060792 484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.061025 484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5077bc9033c94db1b29af5c8d1826ebe, State: Initialized, Role: FOLLOWER
I20260812 11:15:56.061386 484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.062783 484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:56.062816 440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:15:56.062911 440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:56.064042 624 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.064255 624 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7800b83504c39b8b04be434453406, State: Initialized, Role: FOLLOWER
I20260812 11:15:56.064350 440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.064631 624 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.065176 440 leader_election.cc:288] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [CANDIDATE]: Term 1 election: Requested vote from peers a9a7800b83504c39b8b04be434453406 (127.23.97.253:36747), a8aea5bd77df4ac1a5b2948633f48537 (127.23.97.252:33247)
I20260812 11:15:56.065701 770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.065914 770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.066393 624 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:56.066435 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5077bc9033c94db1b29af5c8d1826ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9a7800b83504c39b8b04be434453406"
I20260812 11:15:56.066681 583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:56.066823 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: No bootstrap required, opened a new log
I20260812 11:15:56.065094 778 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.067883 778 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.067950 583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5077bc9033c94db1b29af5c8d1826ebe in term 1.
I20260812 11:15:56.069579 397 leader_election.cc:302] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5077bc9033c94db1b29af5c8d1826ebe, a9a7800b83504c39b8b04be434453406; no voters:
I20260812 11:15:56.069864 770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:56.070093 770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [term 1 LEADER]: Becoming Leader. State: Replica: 5077bc9033c94db1b29af5c8d1826ebe, State: Running, Role: LEADER
I20260812 11:15:56.070444 770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.071419 753 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.071636 753 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8aea5bd77df4ac1a5b2948633f48537, State: Initialized, Role: FOLLOWER
I20260812 11:15:56.071923 753 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } }
I20260812 11:15:56.072232 770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5077bc9033c94db1b29af5c8d1826ebe. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.073232 753 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:56.070019 364 master_runner.cc:186] Error getting master registration for 127.23.97.253:36747: OK, instance_id { permanent_uuid: "a9a7800b83504c39b8b04be434453406" instance_seqno: 1786533355785142 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:15:56.074061 364 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:15:56.074442 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5077bc9033c94db1b29af5c8d1826ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8aea5bd77df4ac1a5b2948633f48537"
I20260812 11:15:56.074687 709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:56.072343 770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:56.075968 709 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5077bc9033c94db1b29af5c8d1826ebe in term 1.
W20260812 11:15:56.076530 810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:56.076606 810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:56.077387 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.077509 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 11:15:56.079124 787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:56.079201 787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:56.077225 495 master_runner.cc:186] Error getting master registration for 127.23.97.252:33247: OK, instance_id { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" instance_seqno: 1786533356004871 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:15:56.079741 495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:15:56.091277 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.072171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.23.97.253:36747)
0812 11:15:56.072274 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.072278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.079837 (+ 7559us) server_negotiation.cc:407] Connection header received
0812 11:15:56.080361 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.080388 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.080430 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.080490 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.082620 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.083373 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.084565 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.084891 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.084965 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.086645 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.086650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.086652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.086678 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.089040 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.089782 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.089786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.089788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.089884 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.090216 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.090220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.090222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.090454 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.090558 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.090978 (+ 420us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.091108 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:15:56.092204 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.071433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.23.97.254:34021)
0812 11:15:56.071557 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.071562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.079521 (+ 7959us) server_negotiation.cc:407] Connection header received
0812 11:15:56.079661 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.079666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.079720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.079809 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.081032 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.081812 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.084071 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.084429 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.084513 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.086493 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.086497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.086500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.086526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.088669 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.090267 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.090271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.090272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.090384 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.090815 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.090818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.090820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.091036 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.091149 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.091944 (+ 795us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.092056 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 11:15:56.095341 848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:56.095427 848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:56.095747 846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:56.096439 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:56.072863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.23.97.252:33247)
0812 11:15:56.074204 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.074208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.080070 (+ 5862us) server_negotiation.cc:407] Connection header received
0812 11:15:56.080227 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.080231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.080280 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.080355 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.084484 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.085204 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.086449 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:56.086825 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:56.086914 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:56.091510 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:56.091514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:56.091517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:56.091543 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:56.094266 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.094957 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.094961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.094963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.095055 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:56.095395 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:56.095399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:56.095401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:56.095626 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:56.095741 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:56.096117 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 11:15:56.096252 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264}
I20260812 11:15:56.096791 846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:56.102439 709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [term 1 FOLLOWER]: Refusing update from remote peer 5077bc9033c94db1b29af5c8d1826ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:56.102531 583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [term 1 FOLLOWER]: Refusing update from remote peer 5077bc9033c94db1b29af5c8d1826ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:56.103219 770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:56.103917 770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:56.107702 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5077bc9033c94db1b29af5c8d1826ebe. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.107834 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.107824 778 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5077bc9033c94db1b29af5c8d1826ebe. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.107944 778 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.115214 770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.115355 770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:56.115484 850 mvcc.cc:205] Tried to move back new op lower bound from 7317640626587500544 to 7317640626468130816. Current Snapshot: MvccSnapshot[applied={T|T < 7317640626587500544}]
I20260812 11:15:56.116756 778 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.116875 778 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.119709 852 mvcc.cc:205] Tried to move back new op lower bound from 7317640626587500544 to 7317640626468130816. Current Snapshot: MvccSnapshot[applied={T|T < 7317640626587500544}]
I20260812 11:15:56.119817 815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.119944 815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:56.121004 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5077bc9033c94db1b29af5c8d1826ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5077bc9033c94db1b29af5c8d1826ebe" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 34021 } } peers { permanent_uuid: "a9a7800b83504c39b8b04be434453406" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 36747 } } peers { permanent_uuid: "a8aea5bd77df4ac1a5b2948633f48537" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 33247 } } }
I20260812 11:15:56.121120 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:56.121590 851 mvcc.cc:205] Tried to move back new op lower bound from 7317640626587500544 to 7317640626468130816. Current Snapshot: MvccSnapshot[applied={T|T < 7317640626587500544}]
I20260812 11:15:56.129698 846 catalog_manager.cc:1347] Generated new cluster ID: 9738dc90cd8249149908cf745e0c6b38
I20260812 11:15:56.129782 846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:56.154500 846 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:56.155730 846 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:56.177397 846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5077bc9033c94db1b29af5c8d1826ebe: Generated new TSK 0
I20260812 11:15:56.178272 846 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:15:56.211994 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.209547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.23.97.253:36747)
0812 11:15:56.209670 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.209675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.210872 (+ 1197us) server_negotiation.cc:407] Connection header received
0812 11:15:56.211092 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.211096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.211148 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.211203 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.211862 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45060: BlockingRecv error: recv got EOF from 127.0.0.1:45060 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 11:15:56.213435 761 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.210238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.23.97.252:33247)
0812 11:15:56.210348 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.210352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.212520 (+ 2168us) server_negotiation.cc:407] Connection header received
0812 11:15:56.212688 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.212691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.212737 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.212798 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.213318 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43514: BlockingRecv error: recv got EOF from 127.0.0.1:43514 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 11:15:56.215369 489 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.214185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.23.97.254:34021)
0812 11:15:56.214328 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.214333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.214471 (+ 138us) server_negotiation.cc:407] Connection header received
0812 11:15:56.214647 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.214651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.214699 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.214762 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.215253 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56870: BlockingRecv error: recv got EOF from 127.0.0.1:56870 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 11:15:56.219781 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.218283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.23.97.253:36747)
0812 11:15:56.218384 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.218387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.218840 (+ 453us) server_negotiation.cc:407] Connection header received
0812 11:15:56.219054 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.219057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.219130 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.219182 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.219667 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45064: BlockingRecv error: recv got EOF from 127.0.0.1:45064 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 11:15:56.224835 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.223240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.23.97.253:36747)
0812 11:15:56.223338 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.223342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.223872 (+ 530us) server_negotiation.cc:407] Connection header received
0812 11:15:56.224054 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.224057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.224106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.224155 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.224736 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45076: BlockingRecv error: recv got EOF from 127.0.0.1:45076 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 11:15:56.230334 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.228718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.23.97.253:36747)
0812 11:15:56.228827 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.228831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.229373 (+ 542us) server_negotiation.cc:407] Connection header received
0812 11:15:56.229584 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.229587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.229649 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.229709 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.230239 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45086: BlockingRecv error: recv got EOF from 127.0.0.1:45086 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 11:15:56.237876 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.236307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.23.97.253:36747)
0812 11:15:56.236464 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.236468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.236999 (+ 531us) server_negotiation.cc:407] Connection header received
0812 11:15:56.237175 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.237178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.237221 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.237269 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.237785 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45088: BlockingRecv error: recv got EOF from 127.0.0.1:45088 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260812 11:15:56.249732 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.247984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.23.97.253:36747)
0812 11:15:56.248096 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.248102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.248720 (+ 618us) server_negotiation.cc:407] Connection header received
0812 11:15:56.248941 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.248945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.248997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.249052 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.249625 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45092: BlockingRecv error: recv got EOF from 127.0.0.1:45092 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 11:15:56.270192 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.268224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.23.97.253:36747)
0812 11:15:56.268350 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.268355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.269034 (+ 679us) server_negotiation.cc:407] Connection header received
0812 11:15:56.269277 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.269281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.269342 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.269412 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.270069 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 11:15:56.306118 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.304719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.23.97.253:36747)
0812 11:15:56.304862 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.304867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.305148 (+ 281us) server_negotiation.cc:407] Connection header received
0812 11:15:56.305326 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.305330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.305388 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.305481 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.305975 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45096: BlockingRecv error: recv got EOF from 127.0.0.1:45096 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 11:15:56.373732 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.372333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.23.97.253:36747)
0812 11:15:56.372486 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.372490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.372778 (+ 288us) server_negotiation.cc:407] Connection header received
0812 11:15:56.372957 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.372961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.373020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.373123 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.373579 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45112: BlockingRecv error: recv got EOF from 127.0.0.1:45112 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:15:56.505435 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.504059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.23.97.253:36747)
0812 11:15:56.504260 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.504265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.504423 (+ 158us) server_negotiation.cc:407] Connection header received
0812 11:15:56.504635 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.504640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.504690 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.504787 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.505291 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45114: BlockingRecv error: recv got EOF from 127.0.0.1:45114 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260812 11:15:56.765264 634 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:56.763876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.23.97.253:36747)
0812 11:15:56.764395 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:56.764402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:56.764421 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:56.764488 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:56.764490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:56.764539 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:56.764630 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:56.765165 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45120: BlockingRecv error: recv got EOF from 127.0.0.1:45120 (error 108)
Metrics: {"server-negotiator.queue_time_us":381}
I20260812 11:15:57.077919 810 catalog_manager.cc:1259] Loaded cluster ID: 9738dc90cd8249149908cf745e0c6b38
I20260812 11:15:57.078029 810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: loading cluster ID for follower catalog manager: success
I20260812 11:15:57.079713 810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: acquiring CA information for follower catalog manager: success
I20260812 11:15:57.080909 810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9a7800b83504c39b8b04be434453406: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:15:57.096604 848 catalog_manager.cc:1259] Loaded cluster ID: 9738dc90cd8249149908cf745e0c6b38
I20260812 11:15:57.096668 848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: loading cluster ID for follower catalog manager: success
I20260812 11:15:57.098188 848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: acquiring CA information for follower catalog manager: success
I20260812 11:15:57.099267 848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8aea5bd77df4ac1a5b2948633f48537: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:15:57.283420 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.279516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.23.97.253:36747)
0812 11:15:57.279804 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.279808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.279824 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:15:57.280013 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.280015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.280147 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.280252 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:57.280503 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.281033 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.281861 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.282078 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.282153 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.282514 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:57.282953 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:57.283059 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:57.283118 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 11:15:57.283188 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260812 11:15:57.290096 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.286485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.23.97.252:33247)
0812 11:15:57.286712 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.286715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.286728 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:57.286867 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.286869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.286972 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.287067 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:57.287357 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.287906 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.288872 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.289089 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.289169 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.289447 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:57.289805 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:57.289891 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:57.289947 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 11:15:57.289988 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260812 11:15:57.297050 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.292869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.23.97.253:36747)
0812 11:15:57.292994 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.292999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.293189 (+ 190us) server_negotiation.cc:407] Connection header received
0812 11:15:57.293544 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.293547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.293601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.293693 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:57.294010 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.294772 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.295865 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.296090 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.296156 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.296582 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:57.296653 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:57.296731 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:57.296865 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 11:15:57.296928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:15:57.297051 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.293291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.23.97.252:33247)
0812 11:15:57.293411 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.293415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.293650 (+ 235us) server_negotiation.cc:407] Connection header received
0812 11:15:57.293742 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.293745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.293796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.293868 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:57.294133 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.294854 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.295883 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.296210 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.296312 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.296579 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:57.296650 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:57.296687 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:57.296873 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 11:15:57.296917 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 11:15:57.298110 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:57.293025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.23.97.254:34021)
0812 11:15:57.293264 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:57.293269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:57.293525 (+ 256us) server_negotiation.cc:407] Connection header received
0812 11:15:57.293625 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:57.293628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:57.293777 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:57.293912 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:15:57.294278 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.294785 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.296459 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:57.296850 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:57.296946 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:57.297368 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:15:57.297640 (+ 272us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:15:57.297736 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:57.297895 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 11:15:57.297953 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260812 11:15:57.300902 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 364
I20260812 11:15:57.319650 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 495
I20260812 11:15:57.332238 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 635
2026-08-12T11:15:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3376 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:15:57 dist-test-slave-zpvx krb5kdc[1015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:15:57 dist-test-slave-zpvx krb5kdc[1015](info): set up 2 sockets
Aug 12 11:15:57 dist-test-slave-zpvx krb5kdc[1015](info): commencing operation
krb5kdc: starting...
W20260812 11:15:58.746227 23943 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/test-user.keytab.
Aug 12 11:15:58 dist-test-slave-zpvx krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.97.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
Entry for principal kudu/127.23.97.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab.
I20260812 11:15:58.869580 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46439
--webserver_interface=127.23.97.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:15:58.981119 1038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:58.981429 1038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:58.981482 1038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:58.981552 1038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:58.985406 1038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:58.985487 1038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:58.985526 1038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:58.985564 1038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:58.985601 1038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:58.990617 1038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.254.keytab
--principal=kudu/127.23.97.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.254:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.97.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:58.991647 1038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:58.992671 1038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:59.000937 1058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:59.000983 1052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:59.000986 1060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:59.001174 1038 server_base.cc:1034] running on GCE node
I20260812 11:15:59.001849 1038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:59.002907 1038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:59.004101 1038 hybrid_clock.cc:648] HybridClock initialized: now 1786533359004083 us; error 38 us; skew 500 ppm
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:59.007095 1038 init.cc:377] Logged in from keytab as kudu/127.23.97.254@KRBTEST.COM (short username kudu)
I20260812 11:15:59.007980 1038 webserver.cc:460] Webserver started at http://127.23.97.254:38273/ using document root <none> and password file <none>
I20260812 11:15:59.008610 1038 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:59.008711 1038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:59.008947 1038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:59.010960 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data/instance:
uuid: "6253b544dbb64eecab14a4e0201c649c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "f096bffdfd6f0f16d5909b1283d7720e"
server_key_iv: "ce1dd8780a0e78f3d1f7d2c2efcfde7b"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.011538 1038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance:
uuid: "6253b544dbb64eecab14a4e0201c649c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "f096bffdfd6f0f16d5909b1283d7720e"
server_key_iv: "ce1dd8780a0e78f3d1f7d2c2efcfde7b"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.015297 1038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:15:59.017807 1071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:59.019043 1038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 11:15:59.019207 1038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal
uuid: "6253b544dbb64eecab14a4e0201c649c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "f096bffdfd6f0f16d5909b1283d7720e"
server_key_iv: "ce1dd8780a0e78f3d1f7d2c2efcfde7b"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.019358 1038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:59.053539 1038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:59.057251 1038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:59.057456 1038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:59.065109 1038 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.254:46439
I20260812 11:15:59.065428 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.254:46439 every 8 connection(s)
I20260812 11:15:59.066260 1038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/data/info.pb
I20260812 11:15:59.069348 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:59.073375 1151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.076112 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1038
I20260812 11:15:59.076237 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-0/wal/instance
I20260812 11:15:59.076565 23943 external_mini_cluster.cc:1428] Setting key dabc95d7d745253cffbab138a9fd5824
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.253@KRBTEST.COM" created.
I20260812 11:15:59.090642 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.076785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.23.97.254:46439)
0812 11:15:59.077110 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.077124 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.077175 (+ 51us) server_negotiation.cc:407] Connection header received
0812 11:15:59.077849 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.077854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.078052 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.078271 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.079227 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.080185 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.081384 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.082054 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.082231 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.085242 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.085252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.085272 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.085304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.087382 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.087906 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.087918 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.087921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.088040 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.088354 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.088357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.088358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.088818 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.088938 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.089479 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.089945 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":254,"server-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260812 11:15:59.090643 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.075760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46439 (local address 127.0.0.1:43196)
0812 11:15:59.076412 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.076474 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.077502 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.078271 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.078277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.078793 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.079048 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.079062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.080345 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.080349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.081212 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.081222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.082267 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.082276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.082435 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.083115 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.083146 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.085061 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.087491 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.087499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.087507 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.087752 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.088134 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.088137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.088138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.088257 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.088962 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.088972 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.089180 (+ 208us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.089334 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.089687 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260812 11:15:59.093622 1151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.094401 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.094144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36258)
0812 11:15:59.094262 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.094347 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 11:15:59.094694 1074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
I20260812 11:15:59.095628 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.095425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36264)
0812 11:15:59.095537 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.095579 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 11:15:59.095901 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:43073: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
Entry for principal kudu/127.23.97.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab.
I20260812 11:15:59.108220 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:43073
--webserver_interface=127.23.97.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:59.145160 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } attempt: 1
I20260812 11:15:59.146171 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.145859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36274)
0812 11:15:59.146025 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.146091 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:15:59.146934 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.146708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36286)
0812 11:15:59.146824 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.146878 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 11:15:59.147074 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:43073: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
I20260812 11:15:59.215247 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } attempt: 2
I20260812 11:15:59.216130 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.215822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36290)
0812 11:15:59.215980 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.216053 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 11:15:59.216925 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.216696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36292)
0812 11:15:59.216808 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.216860 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:15:59.217067 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.253:43073: Network error: Client connection negotiation failed: client connection to 127.23.97.253:43073: connect: Connection refused (error 111)
W20260812 11:15:59.226228 1159 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:59.226526 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:59.226590 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:59.226666 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:59.230666 1159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:59.230742 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:59.230788 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:59.230845 1159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:59.230882 1159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:59.236033 1159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.253.keytab
--principal=kudu/127.23.97.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.253:43073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.97.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:59.237092 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:59.238090 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:59.246474 1170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:59.246587 1159 server_base.cc:1034] running on GCE node
W20260812 11:15:59.246508 1172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:59.246471 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:59.247334 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:59.248312 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:59.249548 1159 hybrid_clock.cc:648] HybridClock initialized: now 1786533359249535 us; error 26 us; skew 500 ppm
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:59.252327 1159 init.cc:377] Logged in from keytab as kudu/127.23.97.253@KRBTEST.COM (short username kudu)
I20260812 11:15:59.253288 1159 webserver.cc:460] Webserver started at http://127.23.97.253:34033/ using document root <none> and password file <none>
I20260812 11:15:59.253854 1159 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:59.253949 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:59.254207 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:59.256206 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data/instance:
uuid: "4ea935fee20849778ff2ef35a40f1d5c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "0325df988f34bc34961fdf8c05cbdb9d"
server_key_iv: "8c413a8ebafdaee0ef24eb9653e57fa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.256867 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal/instance:
uuid: "4ea935fee20849778ff2ef35a40f1d5c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "0325df988f34bc34961fdf8c05cbdb9d"
server_key_iv: "8c413a8ebafdaee0ef24eb9653e57fa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.260511 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:15:59.262828 1189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:59.264027 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:59.264173 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal
uuid: "4ea935fee20849778ff2ef35a40f1d5c"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "0325df988f34bc34961fdf8c05cbdb9d"
server_key_iv: "8c413a8ebafdaee0ef24eb9653e57fa2"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.264290 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:59.285772 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:59.288794 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:59.289002 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:59.296027 1159 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.253:43073
I20260812 11:15:59.296345 1275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.253:43073 every 8 connection(s)
I20260812 11:15:59.297065 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/data/info.pb
I20260812 11:15:59.299783 1278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:59.303500 1278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.304898 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1159
I20260812 11:15:59.305025 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-1/wal/instance
I20260812 11:15:59.305364 23943 external_mini_cluster.cc:1428] Setting key 290ff5b2a51e961ebc35f5a62fe1f1b7
I20260812 11:15:59.308234 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } attempt: 3
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.97.252@KRBTEST.COM; defaulting to no policy
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.97.252@KRBTEST.COM" created.
I20260812 11:15:59.319533 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.308729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36306)
0812 11:15:59.308913 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.308930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.309019 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.309835 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.309838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.310032 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.310145 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.310152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.311716 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.311720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.312927 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.312939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.313821 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.313825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.313934 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.314420 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.314436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.315856 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.317861 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.317866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.317868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.318071 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.318429 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.318432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.318434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.318529 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.318987 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.318990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.319085 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.319208 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.319360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 11:15:59.320019 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.305380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.23.97.254:46439)
0812 11:15:59.305567 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.305570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.306545 (+ 975us) server_negotiation.cc:407] Connection header received
0812 11:15:59.307468 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.307472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.307536 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.307647 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.309092 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.309630 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.311097 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.311345 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.311461 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.314671 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.314677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.314684 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.314728 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.316502 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.317122 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.317128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.317130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.317234 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.317603 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.317608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.317610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.317870 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.317984 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.319269 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.319886 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":478,"server-negotiator.queue_time_us":87}
I20260812 11:15:59.320689 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.308825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.23.97.253:43073)
0812 11:15:59.309364 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.309377 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.309403 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:15:59.309450 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.309458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.309647 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.309821 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.310274 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.311568 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.313075 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.313629 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.313819 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.316006 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.316015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.316031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.316052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.317730 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.318196 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.318209 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.318211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.318319 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.318636 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.318641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.318643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.318874 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.319019 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.319316 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.319637 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":424,"thread_start_us":90,"threads_started":1}
I20260812 11:15:59.320739 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.305856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46439 (local address 127.0.0.1:43208)
0812 11:15:59.306380 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.306422 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.307257 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.307823 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.307830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.308170 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.308873 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.308913 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.309800 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.309805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.310971 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.310982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.311500 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.311504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.311727 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.312427 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.312448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.314499 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.316603 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.316614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.316625 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.317028 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.317348 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.317351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.317353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.317501 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.317986 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.317993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.318284 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.319360 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.320531 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":877,"thread_start_us":143,"threads_started":1}
I20260812 11:15:59.322358 1278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.323199 1151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.323969 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.323667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47098)
0812 11:15:59.323839 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.323898 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 11:15:59.324767 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.324529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47104)
0812 11:15:59.324660 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.324720 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:15:59.324923 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:15:59.331566 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.322880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36310)
0812 11:15:59.323011 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.323027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.323109 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.323531 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.323534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.323706 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.323836 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.323845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.324998 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.325003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.325962 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.325976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.326443 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.326447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.326575 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.326982 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.326999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.328256 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.330082 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.330085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.330087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.330335 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.330619 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.330622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.330624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.330707 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.331077 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.331080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.331159 (+ 79us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.331275 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.331407 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 11:15:59.331693 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.323192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.23.97.253:43073)
0812 11:15:59.323316 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.323320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.323333 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:59.323365 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.323367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.323405 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.323486 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.323973 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.324859 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.326094 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.326325 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.326407 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.328412 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.328417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.328422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.328444 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.329948 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.330441 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.330446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.330448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.330523 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.330795 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.330800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.330802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.330982 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.331086 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.331332 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.331462 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":28}
I20260812 11:15:59.332435 1278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.333066 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.332811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47110)
0812 11:15:59.332934 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.333007 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:15:59.333660 1196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
I20260812 11:15:59.334496 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.334288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47126)
0812 11:15:59.334395 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.334440 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:15:59.334627 1278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
Entry for principal kudu/127.23.97.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab.
I20260812 11:15:59.335709 23943 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
/tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:34283
--webserver_interface=127.23.97.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.97.212:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:15:59.356081 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 1
I20260812 11:15:59.356894 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.356595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47132)
0812 11:15:59.356747 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.356813 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 11:15:59.357674 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.357453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47144)
0812 11:15:59.357560 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.357613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 11:15:59.357838 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
I20260812 11:15:59.383876 1278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 1
I20260812 11:15:59.384720 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.384450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47156)
0812 11:15:59.384584 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.384649 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:15:59.385522 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.385309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47162)
0812 11:15:59.385417 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.385472 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:15:59.385710 1278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
I20260812 11:15:59.433043 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 2
I20260812 11:15:59.434397 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.433967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47172)
0812 11:15:59.434214 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.434295 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260812 11:15:59.435281 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.435014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47186)
0812 11:15:59.435143 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.435205 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:15:59.435477 1151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
W20260812 11:15:59.449221 1295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:15:59.449514 1295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:15:59.449590 1295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:15:59.449680 1295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:15:59.453735 1295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:15:59.453809 1295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:15:59.453846 1295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:15:59.453902 1295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:15:59.453944 1295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:15:59.453890 1278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 2
I20260812 11:15:59.454684 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.454423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47194)
0812 11:15:59.454553 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.454607 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:15:59.455390 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.455170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47210)
0812 11:15:59.455284 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.455336 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:15:59.455503 1278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.97.252:34283: Network error: Client connection negotiation failed: client connection to 127.23.97.252:34283: connect: Connection refused (error 111)
I20260812 11:15:59.459460 1295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.97.212:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.97.254:46439,127.23.97.253:43073,127.23.97.252:34283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/krb5kdc/kudu_127.23.97.252.keytab
--principal=kudu/127.23.97.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.97.252:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.97.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 09:58:34 UTC on ccycloud.quasar-qfmjiv.root.comops.site
build id 4582284
I20260812 11:15:59.460502 1295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:15:59.461522 1295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:15:59.469997 1301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:15:59.470059 1304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:59.470203 1295 server_base.cc:1034] running on GCE node
W20260812 11:15:59.470062 1302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:15:59.470816 1295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:15:59.471868 1295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:15:59.473050 1295 hybrid_clock.cc:648] HybridClock initialized: now 1786533359473019 us; error 46 us; skew 500 ppm
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:15:59.475840 1295 init.cc:377] Logged in from keytab as kudu/127.23.97.252@KRBTEST.COM (short username kudu)
I20260812 11:15:59.476743 1295 webserver.cc:460] Webserver started at http://127.23.97.252:36667/ using document root <none> and password file <none>
I20260812 11:15:59.477303 1295 fs_manager.cc:359] Metadata directory not provided
I20260812 11:15:59.477398 1295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:15:59.477627 1295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:15:59.479589 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data/instance:
uuid: "67f3f2c94b274b83ac3ddacf6b78c781"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "22c412b3506e701200db6d24953fff94"
server_key_iv: "33feb64b40e802383b49450ff39f92cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.480167 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal/instance:
uuid: "67f3f2c94b274b83ac3ddacf6b78c781"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "22c412b3506e701200db6d24953fff94"
server_key_iv: "33feb64b40e802383b49450ff39f92cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.483771 1295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:15:59.486279 1316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:15:59.487577 1295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:15:59.487738 1295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal
uuid: "67f3f2c94b274b83ac3ddacf6b78c781"
format_stamp: "Formatted at 2026-08-12 11:15:59 on dist-test-slave-zpvx"
server_key: "22c412b3506e701200db6d24953fff94"
server_key_iv: "33feb64b40e802383b49450ff39f92cf"
server_key_version: "encryptionkey@0"
I20260812 11:15:59.487855 1295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:15:59.506412 1295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:15:59.509711 1295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:15:59.509917 1295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:15:59.516871 1295 rpc_server.cc:307] RPC server started. Bound to: 127.23.97.252:34283
I20260812 11:15:59.517174 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.97.252:34283 every 8 connection(s)
I20260812 11:15:59.517858 1295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/data/info.pb
I20260812 11:15:59.520684 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:15:59.521890 23943 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu as pid 1295
I20260812 11:15:59.522028 23943 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWFrLZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533060973827-23943-0/minicluster-data/master-2/wal/instance
I20260812 11:15:59.522394 23943 external_mini_cluster.cc:1428] Setting key 08ee38997a445a382af1470ebf15d5be
I20260812 11:15:59.525588 1422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:59.534663 1151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 3
I20260812 11:15:59.538486 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.528192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.23.97.254:46439)
0812 11:15:59.528334 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.528341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.528360 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:59.528429 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.528432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.528471 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.528533 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.529377 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.530139 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.531553 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.531764 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.531848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.534280 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.534283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.534285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.534308 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.536162 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.536829 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.536833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.536835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.536925 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.537402 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.537405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.537407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.537698 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.537808 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.538171 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.538288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260812 11:15:59.538766 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.527466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.254:46439 (local address 127.0.0.1:43218)
0812 11:15:59.527927 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.527976 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.528114 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.528616 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.528622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.529052 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.529221 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.529229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.530323 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.530327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.531404 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.531419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.531923 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.531930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.532029 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.532564 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.532579 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.534118 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.536303 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.536317 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.536324 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.536698 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.537028 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.537033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.537035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.537155 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.537825 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.537831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.537936 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.538078 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.538238 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":21,"thread_start_us":112,"threads_started":1}
I20260812 11:15:59.539000 1428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.523887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.23.97.252:34283)
0812 11:15:59.524289 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.524308 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.524355 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:15:59.525219 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.525242 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.525664 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.526028 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.526740 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.528260 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.529502 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.530209 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.530445 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.533172 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.533177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.533186 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.533208 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.534962 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.535464 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.535471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.535477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.535621 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.536120 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.536126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.536128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.536636 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.537426 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.537591 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.537832 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":257,"thread_start_us":95,"threads_started":1}
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.254@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
I20260812 11:15:59.540839 1422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.545773 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.535144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47222)
0812 11:15:59.535273 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.535291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.535369 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.536019 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.536022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.536420 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.536531 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.536541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.537516 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.537521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.538345 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.538355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.539050 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.539056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.539148 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.539682 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.539695 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.541010 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.544058 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.544061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.544063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.544314 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.544649 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.544652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.544654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.544750 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.545231 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.545236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.545354 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.545446 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.545546 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":22}
I20260812 11:15:59.546548 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.535451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.23.97.252:34283)
0812 11:15:59.535790 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.535799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.535812 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:15:59.535848 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.535850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.535891 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.535978 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.536692 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.537402 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.538498 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.538899 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.539557 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.541155 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.541159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.541162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.541192 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.543938 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.544441 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.544444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.544446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.544548 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.544860 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.544863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.544865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.545096 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.546199 (+ 1103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.546299 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.546410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":144,"threads_started":1}
I20260812 11:15:59.546662 1278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } attempt: 3
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.254@KRBTEST.COM
I20260812 11:15:59.550196 1151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: Bootstrap starting.
I20260812 11:15:59.551656 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.542028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.23.97.254:46439)
0812 11:15:59.542156 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.542160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.542199 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:15:59.542245 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.542248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.542296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.542384 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.542911 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.543504 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.544444 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.544767 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.544862 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.548017 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.548021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.548023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.548046 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.549708 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.550138 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.550143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.550144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.550227 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.550542 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.550546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.550548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.550779 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.550967 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.551338 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.551453 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:15:59.551945 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.541357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.253:43073 (local address 127.0.0.1:36322)
0812 11:15:59.541629 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.541644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.541719 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.542158 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.542161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.542362 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.542481 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.542488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.543658 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.543665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.544441 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.544450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.545044 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.545048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.545171 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.545701 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.545716 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.547828 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.550202 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.550207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.550212 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.550558 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.550865 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.550869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.550871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.550972 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.551455 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.551461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.551562 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.551679 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.551798 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260812 11:15:59.551980 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.541667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.23.97.253:43073)
0812 11:15:59.541787 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.541794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.541813 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:15:59.541853 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.541855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.541923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.542017 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.542624 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.543503 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.544563 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.544923 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.545028 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.547990 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.547994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.547997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.548025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.550058 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.550671 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.550674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.550676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.550757 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.551081 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.551085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.551087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.551310 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.551405 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.551739 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.551845 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 11:15:59.552938 1422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:15:59.553349 1151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:59.554442 1151 log.cc:824] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:59.557132 1151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: No bootstrap required, opened a new log
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.252@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.97.253@KRBTEST.COM
I20260812 11:15:59.561856 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.553352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47242)
0812 11:15:59.553787 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.553800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.553874 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.554102 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.554105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.554254 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.554310 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.554315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.555278 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.555280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.555868 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.555876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.556423 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.556427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.556504 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.556974 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.556993 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.558322 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.560150 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.560153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.560155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.560407 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.560712 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.560716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.560720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.560812 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.561324 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.561328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.561446 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.561550 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.561697 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":1}
I20260812 11:15:59.562397 1428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.553542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.23.97.252:34283)
0812 11:15:59.553670 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.553674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.553948 (+ 274us) server_negotiation.cc:407] Connection header received
0812 11:15:59.553972 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.553975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.554008 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.554074 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.554422 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.555167 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.555985 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.556274 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.556415 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.558456 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.558460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.558462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.558484 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.560034 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.560521 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.560524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.560527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.560612 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.560920 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.560923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.560924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.561186 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.561287 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.562161 (+ 874us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.562262 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":64}
I20260812 11:15:59.563259 1151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.563623 1151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6253b544dbb64eecab14a4e0201c649c, State: Initialized, Role: FOLLOWER
I20260812 11:15:59.564086 1151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.564647 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.553686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.23.97.253:43073)
0812 11:15:59.553773 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.553777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.553789 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:15:59.553828 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.553833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.553877 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.553951 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.554763 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.555472 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.556596 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.556952 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.557075 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.560158 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.560161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.560164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.560193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.562166 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.562723 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.562727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.562729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.562825 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.563155 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.563159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.563161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.563388 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.563622 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.564278 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.564437 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 11:15:59 dist-test-slave-zpvx krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.97.253@KRBTEST.COM for kudu/127.23.97.252@KRBTEST.COM
I20260812 11:15:59.565860 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: Bootstrap starting.
I20260812 11:15:59.565913 1151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:59.568482 23943 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:15:59.569402 1422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:59.569818 1118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6253b544dbb64eecab14a4e0201c649c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43224
I20260812 11:15:59.570058 1118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:15:59.570173 1118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:59.570367 1422 log.cc:824] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:59.571516 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.547063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.97.252:34283 (local address 127.0.0.1:47228)
0812 11:15:59.547325 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 11:15:59.547338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:15:59.547412 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:15:59.555198 (+ 7786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:15:59.555202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:15:59.555373 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:15:59.555452 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.555457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.561568 (+ 6111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.561571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.562371 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:15:59.562383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.563027 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.563031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:15:59.563151 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.563605 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:15:59.563624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:15:59.565172 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:15:59.569829 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.569832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.569834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.570075 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.570427 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:15:59.570430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:15:59.570432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:15:59.570523 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:15:59.570990 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:15:59.570993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:15:59.571093 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 11:15:59.571203 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:15:59.571347 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260812 11:15:59.571539 1118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
W20260812 11:15:59.571708 1453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:59.571772 1453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:59.572062 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.547287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.23.97.252:34283)
0812 11:15:59.554904 (+ 7617us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.554909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.554923 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:15:59.554962 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.554968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.555020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.560472 (+ 5452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.560571 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.561437 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.562523 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.562895 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.565303 (+ 2408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.565397 (+ 94us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.565400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.565403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.565419 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.569704 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.570223 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.570229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.570233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.570313 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.570646 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.570649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.570650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.570861 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.570991 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.571808 (+ 817us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.571916 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":41}
I20260812 11:15:59.572530 1118 leader_election.cc:288] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [CANDIDATE]: Term 1 election: Requested vote from peers 4ea935fee20849778ff2ef35a40f1d5c (127.23.97.253:43073), 67f3f2c94b274b83ac3ddacf6b78c781 (127.23.97.252:34283)
I20260812 11:15:59.572683 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.572780 1422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: No bootstrap required, opened a new log
I20260812 11:15:59.572803 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.574985 1278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: Bootstrap starting.
I20260812 11:15:59.576741 1422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.577016 1422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f3f2c94b274b83ac3ddacf6b78c781, State: Initialized, Role: FOLLOWER
I20260812 11:15:59.577350 1422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.578473 1278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: Neither blocks nor log segments found. Creating new log.
I20260812 11:15:59.578653 1422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:59.578691 1368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6253b544dbb64eecab14a4e0201c649c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67f3f2c94b274b83ac3ddacf6b78c781"
I20260812 11:15:59.578886 1368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:59.575582 1038 master_runner.cc:186] Error getting master registration for 127.23.97.253:43073: OK, instance_id { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" instance_seqno: 1786533359294036 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:15:59.579871 1038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:15:59.580389 1368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6253b544dbb64eecab14a4e0201c649c in term 1.
I20260812 11:15:59.580818 1075 leader_election.cc:302] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6253b544dbb64eecab14a4e0201c649c, 67f3f2c94b274b83ac3ddacf6b78c781; no voters:
I20260812 11:15:59.581112 1434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:15:59.581332 1434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [term 1 LEADER]: Becoming Leader. State: Replica: 6253b544dbb64eecab14a4e0201c649c, State: Running, Role: LEADER
I20260812 11:15:59.581408 1278 log.cc:824] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:15:59.581640 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.581683 1434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.581773 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.583523 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6253b544dbb64eecab14a4e0201c649c. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.583662 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:59.584287 1278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: No bootstrap required, opened a new log
W20260812 11:15:59.584676 1491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:59.584769 1491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:59.590075 1278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.590281 1278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea935fee20849778ff2ef35a40f1d5c, State: Initialized, Role: FOLLOWER
I20260812 11:15:59.590571 1278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } }
I20260812 11:15:59.592013 1278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:15:59.592108 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6253b544dbb64eecab14a4e0201c649c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ea935fee20849778ff2ef35a40f1d5c"
I20260812 11:15:59.592341 1235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:15:59.593875 1235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6253b544dbb64eecab14a4e0201c649c in term 1.
I20260812 11:15:59.594100 1502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:15:59.588280 1295 master_runner.cc:186] Error getting master registration for 127.23.97.253:43073: OK, instance_id { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" instance_seqno: 1786533359294036 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:15:59.595144 1295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:15:59.596874 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.585765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.23.97.252:34283)
0812 11:15:59.585873 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.585878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.586876 (+ 998us) server_negotiation.cc:407] Connection header received
0812 11:15:59.587632 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.587635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.587676 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.587723 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.588618 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.589388 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.590551 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.590878 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.590956 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.592982 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.592989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.592992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.593039 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.594881 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.595433 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.595436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.595438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.595528 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.595859 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.595863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.595864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.596065 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.596205 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.596610 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.596724 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 11:15:59.597879 1502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:15:59.603619 1511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.603762 1511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.605993 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.585250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.23.97.253:43073)
0812 11:15:59.585365 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.585370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.586315 (+ 945us) server_negotiation.cc:407] Connection header received
0812 11:15:59.586638 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.586642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.586694 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.586755 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.595936 (+ 9181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.596753 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.599471 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.599830 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.599920 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.601983 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.601987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.601990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.602017 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.603778 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.604457 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.604461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.604463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.604562 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.604940 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.604944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.604946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.605171 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.605294 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.605696 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.605821 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 11:15:59.607309 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:15:59.586212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.23.97.254:46439)
0812 11:15:59.586311 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.586315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.597537 (+ 11222us) server_negotiation.cc:407] Connection header received
0812 11:15:59.597715 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.597718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.597768 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.597823 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.598748 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.599511 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.600936 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:15:59.601266 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:15:59.601347 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:15:59.603501 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:15:59.603504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:15:59.603507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:15:59.603545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:15:59.605377 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.605890 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.605894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.605895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.605986 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:15:59.606334 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:15:59.606338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:15:59.606340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:15:59.606542 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:15:59.606657 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:15:59.607029 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 11:15:59.607156 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 11:15:59.611728 1368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [term 1 FOLLOWER]: Refusing update from remote peer 6253b544dbb64eecab14a4e0201c649c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:59.611958 1235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [term 1 FOLLOWER]: Refusing update from remote peer 6253b544dbb64eecab14a4e0201c649c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:15:59.612401 1492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:15:59.612617 1434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:15:59.615702 1531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:15:59.615785 1531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:15:59.616679 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6253b544dbb64eecab14a4e0201c649c. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.616801 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.617980 1511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6253b544dbb64eecab14a4e0201c649c. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.618132 1511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.626802 1492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.626928 1492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:59.627228 1511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.627362 1511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.627609 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.627714 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c [sys.catalog]: This master's current role is: LEADER
I20260812 11:15:59.628719 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6253b544dbb64eecab14a4e0201c649c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6253b544dbb64eecab14a4e0201c649c" member_type: VOTER last_known_addr { host: "127.23.97.254" port: 46439 } } peers { permanent_uuid: "4ea935fee20849778ff2ef35a40f1d5c" member_type: VOTER last_known_addr { host: "127.23.97.253" port: 43073 } } peers { permanent_uuid: "67f3f2c94b274b83ac3ddacf6b78c781" member_type: VOTER last_known_addr { host: "127.23.97.252" port: 34283 } } }
I20260812 11:15:59.628839 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:15:59.628964 1539 mvcc.cc:205] Tried to move back new op lower bound from 7317640640963072000 to 7317640640850157568. Current Snapshot: MvccSnapshot[applied={T|T < 7317640640963072000}]
I20260812 11:15:59.629207 1535 mvcc.cc:205] Tried to move back new op lower bound from 7317640640963072000 to 7317640640850157568. Current Snapshot: MvccSnapshot[applied={T|T < 7317640640963072000}]
I20260812 11:15:59.629441 1538 mvcc.cc:205] Tried to move back new op lower bound from 7317640640963072000 to 7317640640850157568. Current Snapshot: MvccSnapshot[applied={T|T < 7317640640963072000}]
I20260812 11:15:59.638808 1502 catalog_manager.cc:1347] Generated new cluster ID: 295a7499bae446cfa18afe8a4e70c31f
I20260812 11:15:59.638883 1502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:15:59.719584 1502 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:15:59.721016 1502 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:15:59.735020 1502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6253b544dbb64eecab14a4e0201c649c: Generated new TSK 0
I20260812 11:15:59.737849 1502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:15:59.774049 1154 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.766870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.23.97.254:46439)
0812 11:15:59.767030 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.767035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.772794 (+ 5759us) server_negotiation.cc:407] Connection header received
0812 11:15:59.773044 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.773051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.773124 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.773186 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.773907 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43252: BlockingRecv error: recv got EOF from 127.0.0.1:43252 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 11:15:59.775718 1432 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.767850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.23.97.252:34283)
0812 11:15:59.767987 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.767992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.774631 (+ 6639us) server_negotiation.cc:407] Connection header received
0812 11:15:59.774826 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.774829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.774885 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.774939 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.775555 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47250: BlockingRecv error: recv got EOF from 127.0.0.1:47250 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 11:15:59.777328 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.768711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.23.97.253:43073)
0812 11:15:59.768846 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.768851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.776284 (+ 7433us) server_negotiation.cc:407] Connection header received
0812 11:15:59.776523 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.776527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.776586 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.776644 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.777200 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36352: BlockingRecv error: recv got EOF from 127.0.0.1:36352 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 11:15:59.782073 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.780559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.23.97.253:43073)
0812 11:15:59.780689 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.780694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.781026 (+ 332us) server_negotiation.cc:407] Connection header received
0812 11:15:59.781274 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.781278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.781355 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.781420 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.781960 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36356: BlockingRecv error: recv got EOF from 127.0.0.1:36356 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 11:15:59.787302 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.785827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.23.97.253:43073)
0812 11:15:59.785947 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.785954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.786342 (+ 388us) server_negotiation.cc:407] Connection header received
0812 11:15:59.786545 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.786548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.786603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.786664 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.787208 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36364: BlockingRecv error: recv got EOF from 127.0.0.1:36364 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 11:15:59.793442 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.791702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.23.97.253:43073)
0812 11:15:59.791806 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.791810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.792345 (+ 535us) server_negotiation.cc:407] Connection header received
0812 11:15:59.792639 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.792642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.792698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.792785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.793349 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36372: BlockingRecv error: recv got EOF from 127.0.0.1:36372 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 11:15:59.801434 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.799704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.23.97.253:43073)
0812 11:15:59.799842 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.799846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.800445 (+ 599us) server_negotiation.cc:407] Connection header received
0812 11:15:59.800665 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.800669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.800723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.800780 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.801339 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36378: BlockingRecv error: recv got EOF from 127.0.0.1:36378 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 11:15:59.813680 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.811784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.23.97.253:43073)
0812 11:15:59.811901 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.811905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.812612 (+ 707us) server_negotiation.cc:407] Connection header received
0812 11:15:59.812845 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.812850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.812907 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.812968 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.813569 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36386: BlockingRecv error: recv got EOF from 127.0.0.1:36386 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 11:15:59.833861 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.832549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.23.97.253:43073)
0812 11:15:59.832699 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.832705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.832932 (+ 227us) server_negotiation.cc:407] Connection header received
0812 11:15:59.833045 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.833048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.833123 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.833231 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.833733 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36396: BlockingRecv error: recv got EOF from 127.0.0.1:36396 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 11:15:59.869498 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.868247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.23.97.253:43073)
0812 11:15:59.868456 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.868462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.868548 (+ 86us) server_negotiation.cc:407] Connection header received
0812 11:15:59.868690 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.868693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.868744 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.868850 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.869362 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36398: BlockingRecv error: recv got EOF from 127.0.0.1:36398 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 11:15:59.936918 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:15:59.935680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.23.97.253:43073)
0812 11:15:59.935851 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:15:59.935856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:15:59.935915 (+ 59us) server_negotiation.cc:407] Connection header received
0812 11:15:59.936054 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:15:59.936056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:15:59.936107 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:15:59.936182 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:15:59.936780 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36400: BlockingRecv error: recv got EOF from 127.0.0.1:36400 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:16:00.069047 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:00.067435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.23.97.253:43073)
0812 11:16:00.067579 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.067583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.067925 (+ 342us) server_negotiation.cc:407] Connection header received
0812 11:16:00.068090 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.068093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.068154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.068249 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:00.068935 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36408: BlockingRecv error: recv got EOF from 127.0.0.1:36408 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 11:16:00.328812 1283 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:16:00.327634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.23.97.253:43073)
0812 11:16:00.327796 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.327800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.327834 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:16:00.327985 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.327987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.328036 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.328126 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:16:00.328704 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36418: BlockingRecv error: recv got EOF from 127.0.0.1:36418 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:16:00.586151 1491 catalog_manager.cc:1259] Loaded cluster ID: 295a7499bae446cfa18afe8a4e70c31f
I20260812 11:16:00.586244 1491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: loading cluster ID for follower catalog manager: success
I20260812 11:16:00.587900 1491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: acquiring CA information for follower catalog manager: success
I20260812 11:16:00.589098 1491 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67f3f2c94b274b83ac3ddacf6b78c781: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:16:00.617156 1531 catalog_manager.cc:1259] Loaded cluster ID: 295a7499bae446cfa18afe8a4e70c31f
I20260812 11:16:00.617255 1531 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: loading cluster ID for follower catalog manager: success
I20260812 11:16:00.618899 1531 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: acquiring CA information for follower catalog manager: success
I20260812 11:16:00.620033 1531 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ea935fee20849778ff2ef35a40f1d5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:16:00.847054 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.843074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.23.97.253:43073)
0812 11:16:00.843333 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.843337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.843412 (+ 75us) server_negotiation.cc:407] Connection header received
0812 11:16:00.843540 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.843542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.843675 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.843794 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:00.844072 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.844625 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.845639 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.845863 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.845941 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.846254 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:00.846684 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:00.846790 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:00.846881 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 11:16:00.846929 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
I20260812 11:16:00.853892 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.850164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.23.97.252:34283)
0812 11:16:00.850421 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.850426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.850444 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:16:00.850539 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.850542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.850663 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.850796 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:00.851028 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.851564 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.852418 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.852668 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.852787 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.853093 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:00.853496 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:00.853638 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:00.853682 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 11:16:00.853736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260812 11:16:00.860865 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.856793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.23.97.254:46439)
0812 11:16:00.857423 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.857430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.857447 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:16:00.857539 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.857541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.857693 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.857836 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:00.858086 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.858600 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.859434 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.859664 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.859779 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.860113 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:00.860426 (+ 313us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:00.860527 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:00.860619 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 11:16:00.860679 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":67,"threads_started":1}
I20260812 11:16:00.861562 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.857073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.23.97.253:43073)
0812 11:16:00.857184 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.857188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.857202 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:16:00.857335 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.857339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.857407 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.857456 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:00.857953 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.858742 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.860129 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.860481 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.860585 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.861051 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:00.861126 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:00.861201 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:00.861356 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 11:16:00.861413 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 11:16:00.861812 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:16:00.856936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.23.97.252:34283)
0812 11:16:00.857031 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 11:16:00.857037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:16:00.857442 (+ 405us) server_negotiation.cc:407] Connection header received
0812 11:16:00.857528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:16:00.857530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:16:00.857577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:16:00.857651 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:16:00.857953 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.858733 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.860463 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:16:00.860806 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:16:00.860918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:16:00.861319 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:16:00.861393 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:16:00.861438 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 11:16:00.861628 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 11:16:00.861682 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 11:16:00.870814 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1038
I20260812 11:16:00.886826 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1159
I20260812 11:16:00.893828 23943 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWFrLZ/build/debug/bin/kudu with pid 1295
2026-08-12T11:16:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3573 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6950 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299928 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST